Mounting a Virtual Box “vmdk” image shouldn’t be this hard….

vmdk

And it wasn’t, well, once I learned how to do it properly. Unfortunately it took me an hour to get there! First I monkeyed around with vmware-tools, and vmware-mount, and then I tried kpartx, and all of it was a wash.

Then, after more searching online, I found a guy that made it plain and simple:

sudo modprobe nbd
sudo qemu-nbd -r -c /dev/nbd1 ./NewVirtualDisk1.vmdk

And that was it! Ubuntu automatically mounted it for me and it popped up on my desktop! It’s amazing when you spend hours following other tutorials, installing more and more things, only to find that there was a simple tool or way to do this that was there all along.

Hopefully it saves you the same headache!

Linux – keep it simple.

Advertisements

Finding the real errors in your Android logcat…

So, you built a rom, and something went wrong. It’s stuck at the boot logo, or it does boot up, but doesn’t do “xyz” action, or it crashes right after you get to the home screen. You’ve played this game before, you pull logs, and all you can see is line after line of errors. Where do you start?

This happens to me all the time. It can be really tough trying to figure out which error is the one that brought the system down, but there are a few key tips to help you unlock the real problem.

  • Look for the key phrases, like “beginning of crash”

Typically, this phrase is when things are going downhill fast. Oft, when things are at this point, the following data will repeat itself and may or may not be helpful, as it may or may not be able to finish writing down the issue. Typically, the lines just before and after this phrase are your biggest clue. Here’s an example:

08-14 15:55:22.357 1468 1468 E System : ******************************************
08-14 15:55:22.358 1468 1468 E System : ************ Failure starting system services
08-14 15:55:22.358 1468 1468 E System : java.lang.IllegalStateException: Signature|privileged permissions not in privapp-permissions whitelist: {com.android.systemui: android.permission.NAVIGATION_EDITOR, org.omnirom.omnistyle: android.permission.CHANGE_OVERLAY_PACKAGES, com.android.systemui: android.permission.PACKAGE_USAGE_STATS, com.android.settings: android.permission.ACCESS_FONT_MANAGER, com.android.settings: android.permission.NAVIGATION_EDITOR, com.android.systemui: android.permission.FORCE_STOP_PACKAGES, com.android.launcher3: android.permission.STATUS_BAR}
08-14 15:55:22.358 1468 1468 E System : at com.android.server.pm.permission.PermissionManagerService.systemReady(PermissionManagerService.java:2005)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.pm.permission.PermissionManagerService.access$100(PermissionManagerService.java:89)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.pm.permission.PermissionManagerService$PermissionManagerInternalImpl.systemReady(PermissionManagerService.java:2052)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.pm.PackageManagerService.systemReady(PackageManagerService.java:21423)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.SystemServer.startOtherServices(SystemServer.java:1791)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.SystemServer.run(SystemServer.java:464)
08-14 15:55:22.358 1468 1468 E System : at com.android.server.SystemServer.main(SystemServer.java:309)
08-14 15:55:22.358 1468 1468 E System : at java.lang.reflect.Method.invoke(Native Method)
08-14 15:55:22.358 1468 1468 E System : at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:495)
08-14 15:55:22.358 1468 1468 E System : at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:838)
08-14 15:55:22.359 1468 1468 D SystemServerTiming: MakePackageManagerServiceReady took to complete: 514ms
08-14 15:55:22.359 1468 1468 E Zygote : System zygote died with exception
08-14 15:55:22.359 1468 1468 E Zygote : java.lang.IllegalStateException: Signature|privileged permissions not in privapp-permissions whitelist: {com.android.systemui: android.permission.NAVIGATION_EDITOR, org.omnirom.omnistyle: android.permission.CHANGE_OVERLAY_PACKAGES, com.android.systemui: android.permission.PACKAGE_USAGE_STATS, com.android.settings: android.permission.ACCESS_FONT_MANAGER, com.android.settings: android.permission.NAVIGATION_EDITOR, com.android.systemui: android.permission.FORCE_STOP_PACKAGES, com.android.launcher3: android.permission.STATUS_BAR}
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.pm.permission.PermissionManagerService.systemReady(PermissionManagerService.java:2005)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.pm.permission.PermissionManagerService.access$100(PermissionManagerService.java:89)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.pm.permission.PermissionManagerService$PermissionManagerInternalImpl.systemReady(PermissionManagerService.java:2052)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.pm.PackageManagerService.systemReady(PackageManagerService.java:21423)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.SystemServer.startOtherServices(SystemServer.java:1791)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.SystemServer.run(SystemServer.java:464)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.server.SystemServer.main(SystemServer.java:309)
08-14 15:55:22.359 1468 1468 E Zygote : at java.lang.reflect.Method.invoke(Native Method)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:495)
08-14 15:55:22.359 1468 1468 E Zygote : at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:838)
08-14 15:55:22.360 1468 1468 D AndroidRuntime: Shutting down VM
——— beginning of crash
08-14 15:55:22.361 1468 1468 E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: main
08-14 15:55:22.361 1468 1468 E AndroidRuntime: java.lang.IllegalStateException: Signature|privileged permissions not in privapp-permissions whitelist: {com.android.systemui: android.permission.NAVIGATION_EDITOR, org.omnirom.omnistyle: android.permission.CHANGE_OVERLAY_PACKAGES, com.android.systemui: android.permission.PACKAGE_USAGE_STATS, com.android.settings: android.permission.ACCESS_FONT_MANAGER, com.android.settings: android.permission.NAVIGATION_EDITOR, com.android.systemui: android.permission.FORCE_STOP_PACKAGES, com.android.launcher3: android.permission.STATUS_BAR}
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.pm.permission.PermissionManagerService.systemReady(PermissionManagerService.java:2005)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.pm.permission.PermissionManagerService.access$100(PermissionManagerService.java:89)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.pm.permission.PermissionManagerService$PermissionManagerInternalImpl.systemReady(PermissionManagerService.java:2052)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.pm.PackageManagerService.systemReady(PackageManagerService.java:21423)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.SystemServer.startOtherServices(SystemServer.java:1791)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.SystemServer.run(SystemServer.java:464)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.server.SystemServer.main(SystemServer.java:309)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at java.lang.reflect.Method.invoke(Native Method)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:495)
08-14 15:55:22.361 1468 1468 E AndroidRuntime: at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:838)
08-14 15:55:22.368 1468 1856 V FontService: CurrentFontPreviewDir absolute path = /data/system/theme/font_previews/com.custom.fonts/Inconsolata
08-14 15:55:22.369 1468 1468 I Process : Sending signal. PID: 1468 SIG: 9

Other great phrases to watch for are:

  1. crash
  2. fail
  3. not found
  4. error
  5. fatal

Usually these phrases will be what you are looking for.

I highlighted the “beginning of crash” line in bold and underlined it to make it easier for you to read. Here we see that the preceding 20-ish lines and following 20-ish lines are all about how the system is struggling with a fatal error, because there are permission issues. (I’ll talk more about solving this next post.)

That’s fine and dandy if you have an actual crash, but doesn’t help if you just have errors in the log and something isn’t functioning correctly. Or sometimes you can have a crash, but there is no “tattle-tale” information that shows you what the problem is. You just start going back through all the “E” errors in the log trying to find one that is the culprit.

  • Compare your log to another to sort out minor errors.

This is not always a viable option, but it has been helpful for me in the past. For instance, recently I purchased an Xperia XA2 Ultra. It had a LineageOS Pie build, but no Resurrection Remix Pie, so I decided to make one. RR didn’t finish booting, but there were a lot of errors in the file, and it was difficult to pinpoint which one was the main culprit.

Take for example, this line:

E Sensors : sns_reg_conf_la.c(766):cei_project_id = BY25

Is it a big deal, or not?

So, I took a logcat of the good, booting, functional LineageOS Pie rom, and a logcat of the bad, not booting RR rom, and compared them with diffuse. Turns out that the above line is in both files, so it’s not too big of a concern right now (but I should fix it later).

lin_rr

This can be very tedious, though, as 28 thousand lines is a lot to sort through. Fortunately, Linux has some handy commands to help you with that. First, I cut the time out of the files:

$ cut -c31- linfirstbootedit.txt |tee linNoTime.txt
$ cut -c31- rrFirstBootedit.txt |tee rrNoTime.txt

Now I had the files without the time at the beginning, which makes them easier to sort. Then, I sorted them and compared them, pulling only the different lines.

comm -23 <(sort rrNoTime.txt) <(sort linNoTime.txt) > Just_rr_Uniq.txt

The sort function puts them in alphanumeric order. Once the two files are sorted, they are compared, and I only take the output of the RR log file that is unique. Now I have eliminated all errors that were in both files. Thus, any errors that were in LineageOS were okay for booting (not okay totally, but didn’t prevent function).

Now I have a list of just the unique lines. Now I can just grab the errors:

cat Just_rr_Uniq.txt |grep ^” E”

And viola! Now I am only looking at the errors that are unique to the RR rom. That cut my file from 28000 lines to a simple 1280! Whatever these errors are, they would be a good place to start looking, since they only happen on the rom that doesn’t work properly.

It’s also good to keep your original logcat untouched, so you can reference it later. Note that the grep’d, sorted, and cut file will be missing key information around the error you are looking at. Hopefully that helps you get pointed in the right direction.

Linux – keep it simple.

Home Photo Server, Part 4: Battery Backup

download

One of the problems that I have with a local server is that we frequently have power “blips”. I don’t want to say outages, because the power is on about 99% of the time. The issue that we have is occasional “brown outs” where the power dips, but doesn’t drop out about once a month. We also have noticed about every other week it seems that the power will blink for just a moment.

Usually about three or four times a year the power will actually go out for more than an hour. It’s not uncommon for a minute long outage on a regular basis though. The worst outage we had was about 3 years ago, when the power was out at our house for 4 days! It was winter time, and I had to run the generator to keep the boiler going. Needless to say, we are prepared around here for the power to go out.

And that’s why I wanted my computer to be prepared as well. Anytime you are using something as a server that you want to make sure it is still running, you really should have it on a backup. So, I purchased an APC BackUPS 1350. By the specs, and it’s own digital display, it should keep my server and monitor running for 15 minutes during an outage.

That’s handy, but I wanted to take it one step further. I’m at work during the week day, and at church on Sunday, so I may not be around when the power goes out. If that happens, having it run for 15 minutes and then die really didn’t help me much. Fortunately, I found some great articles, and installed apcaccess to help me automatically control everything. First I added the EPEL repository, and then installed apcaccess:

# yum install epel-release

# yum install apcupsd

Then I navigated to the apc directory to change my settings…

# cd /etc/apcupsd/

# nano apcupsd.conf

This is a bit lengthy, but here is my apcupsd.conf file, without all the comment lines:

## apcupsd.conf v1.1 ##
#
# for apcupsd release 3.14.14 (31 May 2016) – redhat
#
# “apcupsd” POSIX config file

#
# Note that the apcupsd daemon must be restarted in order for changes to
# this configuration file to become active.
#

#
# ========= General configuration parameters ============
#

UPSNAME SRVRAPC
UPSCABLE usb
UPSTYPE usb
DEVICE
POLLTIME 60
LOCKFILE /var/lock
SCRIPTDIR /etc/apcupsd
PWRFAILDIR /etc/apcupsd
NOLOGINDIR /etc
#
# ======== Configuration parameters used during power failures ==========
#
ONBATTERYDELAY 6
BATTERYLEVEL 20
MINUTES 5
TIMEOUT 0
ANNOY 60
ANNOYDELAY 60
NOLOGON disable
KILLDELAY 0
#
# ==== Configuration statements for Network Information Server ====
#
NETSERVER on
NISIP 127.0.0.1
NISPORT 3551
EVENTSFILE /var/log/apcupsd.events
EVENTSFILEMAX 10
#
# ========== Configuration statements used if sharing =============
# a UPS with more than one machine
UPSCLASS standalone
UPSMODE disable
#
# ===== Configuration statements to control apcupsd system logging ========
#
STATTIME 0
STATFILE /var/log/apcupsd.status
LOGSTATS off
DATATIME 0
#FACILITY DAEMON
#
# ========== Configuration statements used in updating the UPS EPROM =========
#
SELFTEST 336

The important parts being that I set MINUTES to 5, meaning that if the UPS believes that it only has 5 minutes of power left at the current usage rate, it will initiate a computer shutdown command to the server computer. Likewise, if the BATTERYLEVEL drops below 20%, it will also initiate a shutdown command (which, theoretically, would be the 3 minute mark). I suppose you don’t have to set both, but I wanted to make sure under either condition, it shut down the server computer gracefully.

A really funny parameter is NETSERVER on and NISIP 127.0.0.1. At first I turned these off, but then found that the USB connection to the UPS didn’t work unless they were on and set to the local interface. So be sure you turn those on if you are using one of these as well.

Be sure to start the service and make sure it’s enabled:

# systemctl stop apcupsd
# systemctl start apcupsd
# systemctl enable apcupsd

After all of that, now I can just type apcaccess to see the UPS status:

APC : 001,036,0880
DATE : 2019-07-01 09:13:37 -0800
HOSTNAME : localhost.localdomain
VERSION : 3.14.14 (31 May 2016) redhat
UPSNAME : SRVRAPC
CABLE : USB Cable
DRIVER : USB UPS Driver
UPSMODE : Stand Alone
STARTTIME: 2019-06-27 14:34:23 -0800
MODEL : Back-UPS NS 1350M2
STATUS : ONLINE
LINEV : 121.0 Volts
LOADPCT : 34.0 Percent
BCHARGE : 100.0 Percent
TIMELEFT : 16.3 Minutes
MBATTCHG : 20 Percent
MINTIMEL : 5 Minutes
MAXTIME : 0 Seconds
SENSE : Medium
LOTRANS : 88.0 Volts
HITRANS : 139.0 Volts
ALARMDEL : No alarm
BATTV : 27.3 Volts
LASTXFER : No transfers since turnon
NUMXFERS : 0
TONBATT : 0 Seconds
CUMONBATT: 0 Seconds
XOFFBATT : N/A
SELFTEST : NO
STATFLAG : 0x05000008
SERIALNO : 3B1907X23168
BATTDATE : 2019-02-13
NOMINV : 120 Volts
NOMBATTV : 24.0 Volts
NOMPOWER : 810 Watts
FIRMWARE : 954.e3 .D USB FW:e3
END APC : 2019-07-01 09:14:06 -0800

I might play with the high/low transfer, it seems a bit to wide of a spectrum to me, but we’ll see how it does. Either way, I feel a lot safer now in terms of power problems for my server.

Linux – keep it simple.

Home Photo Server, Part 3: https, keys, and certificates

In the continuing saga of the Google Photos alternative, by running a home server with Dynamic Domain Name Service (DDNS), very secure FTP (vsftp), and an Apache web server coupled with Piwigo, we’ve automated the entire process of getting the photographs I take with my phone all put on the server and view-able by those we share it with. And, to God be the glory, it even works! But there are still a few more steps to consider, and today I’d like to look at a more secure interface with https.

So, for https, one of the big things you need to have secure http (the language of the web), is a public/private key pair, for assymetric encryption, and a digital certificate, certifying that those keys are in fact yours. There are a couple of ways to do this on the cheap. Of course, you could pay for a domain name and certificate from a company (like VeriSign and others). Or, you can go with a free certificate from Let’s Encrypt. The only problem that I had with them is that they didn’t seem to support DDNS. Perhaps further research is required. But, if you are using a static IP address, with a domain name, then you certainly can do these options.

Another option, no matter what the use, is self generating your certificate. Now, we do have to clarify, most browsers will warn the user that something is wrong when they go to your web page because the certificate is not signed by a “trusted” source. However, since, in my case, my wife and I are the only ones who log in, I think we can trust our own certificate. The benefit to having your own certificate verses not having one is that now we can use https. If we add the certificate as trusted to the browser, then we will be able to know that we are still talking to our own server, as long as the certificate doesn’t change.

First thing we need is mod_ssl:

# yum install mod_ssl

Once that installs, we need to make a directory for our keys and set the permissions:

# mkdir /etc/ssl/private
# chmod 700 /etc/ssl/private

And of course, generate the keys themselves, as well as set Diffie-Hellman parameters on them for more security:

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
# openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048
# cat /etc/ssl/certs/dhparam.pem | tee -a /etc/ssl/certs/apache-selfsigned.crt

Then, we need to tell Apache, our web server, to use our certificates:

# nano /etc/httpd/conf.d/ssl.conf

Edit as appropriate to point to your new keys and certificate. You can also make a forced redirect to have all http traffic be redirected to https:

# nano /etc/httpd/conf.d/non-ssl.conf

In the file, write this, but replace with your server name:

<VirtualHost *:80>
ServerName yourServerName
Redirect “/” “https://yourServerName&#8221;
</VirtualHost>

Now you need to punch a few holes in the firewall for https traffic….

# firewall-cmd –add-service=https
# firewall-cmd –runtime-to-permanent

And restart everything:

# systemctl restart httpd.service

# firewall-cmd –reload

Now you should be able to check out your website via https! Here you can see the warning message I get for my self generated certificate.

httpsPiwigo

Linux – keep it simple.

Home photo server, part 1: Server Setup, SCP and FTP

While there are many, many options out there for photo storage, if you are looking for a home storage solution that does NOT involve just plugging in your phone and dumping the pictures to your hard drive, you have to get a little technical. (By the way, if that is what you have to do, there is no shame in it. It is probably a lot safer doing that than letting Google hold all of your photos.)

The first thing I needed was a server. Granted, you could use just about anything these days, and there are a lot of open source/open hardware type solutions, but I was gifted an older, generation 1 Dell PowerEdge 1950 from a friend. Granted, it was made in 2006, but it still is 64 bit, has two quad core Xeon 2 GHz processors, and I loaded it with 24 GB of ram. You can get them on eBay now for about $60. A little overkill for this sort of thing, but the price was right! As a bonus, it supported hardware raid, and I put two 2 TB drives in a mirror array, so that 2 TB of space that was backed up.

From there I loaded CentOS 7 on it per the usual installation method, and updated the system. I also purchased an APC battery backup unit, a Back-UPS 1350. This would only hold the power on for about 15 minutes, but it would help for brown outs, and frequent “blips” where the power goes out for only a second or so, which is common where I live. Later I’ll have to do a post on setting up the auto-shutdown and controls, because that was rather interesting.

So the next thing I needed, if I wanted this to work, was a domain. I needed a way to contact my home computer from my cell phone, especially while not at home. Granted, you could set all of this up so when you come home your phone would automatically back up your photos, but I wanted to be able to do this from abroad. Thus enter No-ip.com. I’ve used them before, and it is great if you are looking for a cheap, cheap solution. Because it is free.

Granted, a free account your hostname will need to be manually renewed every 30 days, but they send you an email, and all you have to do is click the link to keep it active, so it is pretty easy. After creating an account, logging in, getting a dynamic IP address, then all I had to do was install the DUC software. DUC is the Dynamic Update Client software that allows:

“Remote access your computer, DVR, webcam, security camera or any internet connected device easily. Dynamic DNS points an easy to remember hostname to your dynamic IP address.” (noip.com)

All you have to do is download the source code and compile it. It went like this:

$ cd noip-2.1.9-1/
$ make
$ sudo make install

After entering my password, it ran through an installation script and asked me for my account name, password for the account, and which DDNS I wanted to associate with this computer. It is interesting, you can have several.

From here, it then became a matter of preference on how to continue. I toyed with several options on my Android phone for how to get the photos from the phone to the computer over the internet.  One of the first methods I tried was using scp, or secure copy over ssh. So, I installed ssh on my server.

# yum install ssh-server

# cd /etc
# cd ssh/
# ls
# nano sshd_config

I then edited the sshd_config to my liking, there are a lot of guides on this on the internet, so I wont belabor the point here. I will note that I use non-standard ports as a form of extra security, however slight that may be, so you may consider doing the same, but essentially it works as is once installed. Then I opened the ports in the firewall – I list the standard ports here, for ease of following along:

# firewall-cmd
# firewall-cmd –help\
# firewall-cmd –help
# firewall-cmd –add-port=22/tcp
# firewall-cmd –add-port=22/tcp –permanent

And that worked great. Unfortunately, scp is slow and can be cumbersome from an Android phone, especially since I didn’t find any apps that would sync my directories automatically (that were open source so I knew what was really being sync’ed). However, I found several open source options that would sync automatically via FTP. So I decided to install “very Secure FTP”, or vsftp, like so:

# yum install vsftpd

# cd /etc/vsftpd/

# ls

# nano vsftpd.conf

Again, I set it up to my needs, but you can check out this guide for ideas. I also needed to punch some holes in the firewall for the service and for both active and passive mode, since several Android apps would use either.

# firewall-cmd –add-port=21/tcp
# firewall-cmd –add-port=21/tcp –permanent
# firewall-cmd –add-port=20/tcp
# firewall-cmd –add-port=20/tcp –permanent

# firewall-cmd –permanent –add-port=40000-50000/tcp

And viola! All that was left was a quick restart of the processes:

# firewall-cmd –reload

# systemctl restart vsftpd

And now I could use FTP apps on my Android phone to sync my pictures from my phone automatically to the home server! In case you are wondering, a great app for this is on F-Droid, the open source app repository of open source apps. It is called SyncTool, and it is very handy. It supports FTP sync one way, both ways, automatic scheduling or running jobs manually.
image

Wheeow, that was a long post, but now my photos were being automatically backed up. However, that’s only part of the story, because if I was convincing my wife to ditch Google Photos, I needed to also have a way to browse them online, share them, organize them, etc…. It was time for a web server. Guess we’ll cover that next.

Linux – keep it simple.

Merging Two MP4 Files From The Command Line

term

One of the problems I find with video editing, is sometimes it becomes very complicated to do something very simple. Take my current need. I needed to simply tie two mp4 files together, unedited, just hook one to the next. There are lots of gui types, but that requires opening them up, adjusting settings, loading or importing the files, and just a little more time and work than I want for such a simple task.

Fortunately, I found a really easy to use method to do that from the command line: MP4Box!

You can install it like so:

$ sudo apt-get install gpac

$ sudo ln -s /usr/local/lib64/libgpac.so /usr/lib/libgpac.so

Once installed, it is very, very simple to use:

 

$ MP4Box -cat myFirstVideo.mp4 -cat mySecondVideo.mp4 -new myNewVideoFile.mp4

That’s it! Now, whenever I need to add two videos together, all I have to do is run one simple command, and in seconds, it is done! That’s what I like to call simple!

Linux – keep it simple.

Wondershaper: The wonder of shaping your internet traffic!

wondershaper

I always end my posts with “Linux – keep it simple.” And today is one of those great days when I am using a really well made, simple, Linux tool: Wondershaper.

Wondershaper allows you to “throttle” network traffic on any interface to a desired capped speed. So, for instance, if you need to have one computer upload videos, like I am today, you will probably max out your upload speeds. The problem with this is that your wife/family/friends/office workers, etc., whomever you share internet with, will suddenly be unable to surf the web.

Or perhaps you are syncing the Android source code, and it is taking up all of your download bandwidth, making it impossible for you to watch a movie on your Vudu/Roku/name your device player.

So, if you want to have a simple to use command line tool to restrict or limit traffic on one of your Linux computers, here it is:

# wondershaper eno1 1000 500

That’s it. The name of the program is wodershaper and the arguments are: interface, download speed, and upload speed. In this case, I want wondershaper to try to throttle traffic for that machine on interface “eno1” to be 1000 kb/s down, and 500 kb/s up. Here is the expanation from the man pages for it:

wondershaper [ interface ] [ downlink ] [ uplink ]
Configures the wondershaper on the specified interface, given
the specified downlink speed in kilobits per second, and the
specified uplink speed in kilobits per second.

You do need to run this as root to have access to control the device interfaces, but this program works beautifully. It is especially handy when working with virtual machines that you wish to limit!

Linux – keep it simple.

Bluetooth Breakdown: Sending my own commands with gatttool

Yes, there is purposely one too many t’s in GattTool.

bt_success

Last time we looked at the Bluetooth Low Energy (BLE) packets being sent from my Android app to my home made auto start. Here’s what we saw:

214231313a – Button 1 pressed
214231303b – Button 1 released
2142323139 – Button 2 pressed
214232303a – Button 2 released

With a little more digging, I also found that the “handle” is important, and per the Bluetooth packet breakdown in WireShark, that handle is 0x001a. Further, my particular Bluetooth feather’s MAC address is F4:64:8E:90:D8:C3.

With all of this in mind, I entered gatttool in the interactive mode, but I kept having trouble connecting to my device. With further research, I found an article on WordPress by a guy named Jack. He got me pointed in the right direction!

What I needed to do before connecting was this:

# btmgmt le on
# btmgmt bredr off

To set the computer’s Bluetooth manager to the proper Low Energy reading/writing state. Then I entered interactive mode while monitoring my auto start with a USB serial monitor. Here’s what I typed:

root@alaskalinuxuser-OptiPlex-7010:/home/alaskalinuxuser# gatttool -t random -b F4:64:8E:90:D8:C3 -I
[F4:64:8E:90:D8:C3][LE]> connect
Attempting to connect to F4:64:8E:90:D8:C3
Connection successful
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214231313a
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214231303b
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 2142323139
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214232303a
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214232303a
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214231303b
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 2142323139
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 214232303a
[F4:64:8E:90:D8:C3][LE]> char-write-cmd 0x001a 2142323139
[F4:64:8E:90:D8:C3][LE]> disconnect

(gatttool:22352): GLib-WARNING **: Invalid file descriptor.

[F4:64:8E:90:D8:C3][LE]> disconnect
[F4:64:8E:90:D8:C3][LE]> exit
root@alaskalinuxuser-OptiPlex-7010:/home/alaskalinuxuser#

And here is what I saw on the monitor:

Less than 13v, Pin A3: 0.00
Less than 13v, Pin A3: 0.00
Less than 13v, Pin A3: 0.00
Less than 13v, Pin A3: 0.00
Button 1 released
Less than 13v, Pin A3: 0.00
startTimer 4
Less than 13v, Pin A3: 0.00
startTimer 3
Less than 13v, Pin A3: 0.00
startTimer 2
Less than 13v, Pin A3: 0.00

Success! Great! I was able to enter interactive mode, connect, and send the right commands! This is great!

Then I switched to non-interactive mode, by writing a custom script or two. This is the first one, to start the auto start:

#!/bin/bash

btmgmt le on
btmgmt bredr off
gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214231313a
gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214231303b
#gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 2142323139
#gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214232303a
exit 0

And one to stop the auto start:

#!/bin/bash

btmgmt le on
btmgmt bredr off
#gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214231313a
#gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214231303b
gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 2142323139
gatttool -t random -b F4:64:8E:90:D8:C3 –char-write-req -a 0x001a -n 214232303a
exit 0

And a quick test was successful! Now I can send my commands to start or stop the auto start from the command line with my computer. Now it’s just time to set it up on my Ubuntu Touch phone!

Linux – keep it simple.

Video Tutorial on How to Compile Android and Modify Kernels

video

For those interested, I have just posted a video tutorial series on XDA for building Android Oreo, Nougat, Marshmallow, and Lollipop on 5 different phones, the emulator, and 5 different ROMs. Also included are custom kernel editing, adding apps, changing source code, backgrounds, and more. Here’s what I posted:

From XDA: https://forum.xda-developers.com/android/general/guide-how-to-build-custom-roms-kernel-t3814251

===========================================================================

Praise God! Finally a video tutorial of how to build Android and modify kernels!

I have created a video tutorial and guide for how to compile Android, from Lollipop through Marshmallow, Nougat, and Oreo. The video series covers several different phones, the emulator, kernel and rom editing, app source code editing, and much more!

Who is this video series for?
Well, this video tutorial is a step by step guide built primarily for the beginner. This is written for those who already know how to flash TWRP, CWM, or the like, and who have installed a custom rom before. This is designed to help those who are ready to move up from flashing and installing other peoples custom rom to actually start making their own custom roms. I recommend that a beginner watch the entire series in numerical/alphabetical order (the videos are marked).

That said, I believe that an intermediate developer may find a useful trick here and there, and they should just skip ahead to videos of interest. Perhaps kernel development, or something along those lines.

An advanced rom/kernel developer will probably far exceed my feeble abilities, and will not likely find much useful information here. Perhaps if you are an advanced developer, you would consider continuing the tutorial or making an advanced video series! (See further posts for recommendations on contributing videos.)

Why did you put this together?
Well, after building roms for several different devices, I started receiving requests from users who wanted to start building their own roms, but didn’t know how. I didn’t have enough time to answer everyones questions, so I wrote a few guides, pointed others to guides that were available, but there are some things that you just need to see to understand. Hence, the video tutorial. I just hope that someone finds it useful.

This course was written in order! While Lollipop and Marshmallow are old by today’s standards, there is still good learning value in building them, and there are topics covered there that really make them worth watching.

What’s in the videos?
During the series, we will be building for the emulator, as well as 5 different phones of various brands, and 5 different roms. I hope that this will give the viewer a good idea of how to build for their own specific phone as they see the differences and similarities across the phones and custom roms.

[CODE]
+ Ubuntu installation
+ Java installations
+ Using Git, GitHub, GitKraken, and the command line
+ Fastboot and ADB
+ Heimdall/Odin
+ QFIL, QPST, SALT, and other tools
+ AOSP, SlimRoms, PACrom, AOKP, AOSCP
+ Lollipop, Marshmallow, Nougat, Oreo
+ Errors
+ Overclocking CPU/GPU
+ Adding Governors and I/O Schedulers
+ Sound modifications
+ Changing app colors, text, and icons
+ Adding prebuilt apps
+ Adding source code
+ Converting device from one rom to another
+ AND MORE!
[/CODE]

**** This is an UNOFFICIAL TUTORIAL. Use at your own risk! ****
Download links:
Ogg Vorbis Video GitLab:
[url]https://gitlab.com/alaskalinuxuser/course_android_developer_guide[/url]
Clicking on a video in GitLab will allow you to watch it online.

Ogg Vorbis Video Downloads:
[url]https://gitlab.com/alaskalinuxuser/course_android_developer_guide/-/archive/master/course_android_developer_guide-master.zip[/url]
This download is rather large due to the multiple videos.

MP4 Video GitLab:
[url]https://gitlab.com/alaskalinuxuser/course_android_developer_guide_mp4[/url]
Clicking on a video in GitLab will allow you to watch it online.

MP4 Video Downloads:
[url]https://gitlab.com/alaskalinuxuser/course_android_developer_guide_mp4/-/archive/master/course_android_developer_guide_mp4-master.zip[/url]
This download is rather large due to the multiple videos.

I also have several written guides available on XDA, here are a few:

Building ROMs for the Galaxy Note Edge: [url]https://forum.xda-developers.com/note-edge/general/guide-build-aosp-roms-kernels-note-edge-t3488840[/url]
Building ROMs for the Galaxy S4: [url]https://forum.xda-developers.com/galaxy-s4-tmobile/general/guide-step-step-instructions-building-t3402637[/url]

===========================================================================

Be sure to check out the videos or the XDA thread! I hope that these will help some of the aspiring Android developers out there!

Linux – keep it simple.

How to check your governor and I/O scheduler from the command line

The other day, when working through some issues with kernel governors and I/O schedulers, I realized that I was a bit rusty on where to find them from the command line. After some searching and tinkering, I thought I might save some internet searcher the long and winding road to find this out.

From the command line:

[CODE]
$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
conservative ondemand userspace powersave performance

$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand

$ cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
[/CODE]

It’s a pretty simple trick to just read what is written in the /sys directory. As you can see here, my laptop system is running the ondemand kernel governor with the deadline I/O scheduler. If you are using this for yourself, you can check any of the cpu’s and drives by inputing them instead of the cpu0 and sda which I used here. This works on just about anything running a Linux kernel, from computers to cell phones like Android.

Linux – keep it simple.