Difference between revisions of "Template:Luneos build"

From WebOS-Ports
Jump to navigation Jump to search
(Created page with "<noinclude> {{warning |heading=DO NOT EDIT! |warning=THIS PAGE IS NOT A REGULAR WIKI PAGE. IT IS A WIKI *TEMPLATE* AUTO-INCLUDED INTO POTENTIALLY HUNDREDS OF OTHER PAGES. AN...")
 
(Also require bzr and gcc-multilib.)
 
(42 intermediate revisions by 5 users not shown)
Line 1: Line 1:
<noinclude>
+
== Setup the build environment ==
{{warning
 
|heading=DO NOT EDIT!
 
|warning=THIS PAGE IS NOT A REGULAR WIKI PAGE.  IT IS A WIKI *TEMPLATE* AUTO-INCLUDED INTO POTENTIALLY HUNDREDS OF OTHER PAGES.  ANY CHANGE MADE HERE WILL AFFECT ALL THESE PAGES SIMULTANEOUSLY, INCLUDING PAGES DEALING WITH MANY TYPES OF DEVICES.  SO DO NOT EDIT THIS PAGE UNLESS YOU KNOW '''EXACTLY''' WHAT YOU ARE DOING.  THANKS!}}
 
</noinclude>
 
  
== Build CyanogenMod and ClockworkMod Recovery ==
+
Before you can build, you will need some tools. If you try to build without them, bitbake will fail a sanity check and tell you about what's missing, but not really how to get the missing pieces.
  
 +
If your distribution is Ubuntu, you need to be running the bash shell. You can be sure you are running this shell by entering the following command and selecting "No" at the prompt:
  
===Prepare the Build Environment===
+
    $ sudo dpkg-reconfigure dash
{{Commentlink
+
         
|section=prepare-build-environment-for-{{{devicename}}}
+
The essential and graphical support packages you need for a supported Ubuntu or Debian distribution are shown in the following commands:
}}
+
   
 +
    $ curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
 +
    $ sudo apt-get install gawk wget git-core diffstat unzip texinfo \
 +
    build-essential chrpath libsdl1.2-dev xterm nodejs bzr
  
{{Note
+
(Recently, building on 64-bit Ubuntu 16.04 also required apt-get install gcc-multilib g++-multilib.)
|note=You only need to do these steps the first time you build. If you previously prepared your build environment and have downloaded the CyanogenMod source code for another device, skip to '''[[#Prepare the device-specific code|Prepare the device-specific code]]'''.
 
}}
 
  
 +
For other Linux distributions please have a look at [https://www.yoctoproject.org/docs/1.5/yocto-project-qs/yocto-project-qs.html the Yocto Project Quick Start guide].
  
====Install the SDK====
+
{{note|note=The steps below will setup your build environment for a development build. If you want to build the latest release use ENV_NAME=stable instead but keep in mind that all new developed features need to go into testing first before they are submitted to stable.}}
{{Commentlink
+
<code>
|section=install-the-sdk-for-{{{devicename}}}
+
  $ cd into-your-build-directory
}}
+
  $ mkdir webos-ports-env && cd webos-ports-env
 +
  $ export ENV_NAME=testing
 +
  $ wget https://raw.github.com/webOS-ports/webos-ports-setup/$ENV_NAME/Makefile
 +
  $ make setup-webos-ports
 +
</code>
  
:If you have not previously installed [[Doc:_adb_intro|adb]] and [[Doc:_fastboot_intro|fastboot]], install the [[sdk|Android SDK]]. "SDK" stands for Software Developer Kit, and it includes useful tools that you can use to flash software, look at the system logs in real time, grab screenshots, and more-- all from your computer.
+
This has been tested on Gentoo (shr-chroot) and Ubuntu-12.04/13.04/14.04/15.10/16.04 amd64/x64 and should work almost everywhere where valid toolchain is provided.
  
{{tip|tip=While the SDK contains lots of different things-- the two tools you are most interested in for building Android are [[adb]] and [[fastboot]], located in the <code>/platform-tools</code> directory.}}
+
You need a lot of RAM to link QtWebEngine, make sure you have at least 6GB (it's OK when some of that is swap, because it's used only for short part of build). QtWebEngine needs so much ram to link because it's linking with debug symbols (huge files) which are stripped later in do_package after creating -dbg packages.
  
====Install the Build Packages====
+
== Building ==
{{Commentlink
 
|section=install-the-build-packages-for-{{{devicename}}}
 
}}
 
  
Several "build packages" are needed to build CyanogenMod. You can install these using the package manager of your choice.
+
Make sure you're _not_ root, as the build will fail. If you SU in the middle, don't forget to set up env afresh (. setup-env)
  
{{tip|tip=A [[wikipedia:Package_manager|package manager]] in Linux is a system used to install or remove software (usually originating from the Internet) on your computer. With Ubuntu, you can use the Ubuntu Software Center.  Even better, you may also use the <code>apt-get install</code> command directly in the Terminal.  (Learn more about the [[wikipedia:Advanced_Packaging_Tool|apt]] packaging tool system from [[wikipedia:Advanced_Packaging_Tool|Wikipedia]].)}}
+
To configure to build (notice '.' which is actually bash 'source' command):
  
For both 32-bit & 64-bit systems, you'll need:
+
<code>
 +
  $ cd into-your-build-directory/webos-ports-env/webos-ports
 +
  $ . setup-env
 +
</code>
  
<code>git-core gnupg flex bison gperf libsdl1.2-dev libesd0-dev libwxgtk2.8-dev squashfs-tools build-essential zip curl libncurses5-dev zlib1g-dev openjdk-6-jre openjdk-6-jdk pngcrush schedtool libxml2 libxml2-utils xsltproc</code>
+
To update metadata
  
In addition to the above, for 64-bit systems, get these:
+
<code>
<code>g++-multilib lib32z1-dev lib32ncurses5-dev lib32readline-gplv2-dev gcc-multilib</code>
+
  # go one level up to the webos-ports-env dir
 +
  $ cd ..
 +
  $ make update
 +
  # or if it shows warning about different bblayers.conf or layers.txt
 +
  $ make update-conffiles && make update
 +
  # you can also add UPDATE_CONFFILES_ENABLED = 1 to config.mk
 +
  # if you never want to have any uncommited changes in your checkouts RESET_ENABLED = 1 in config.mk
  
Also see [http://source.android.com/source/initializing.html http://source.android.com/source/initializing.html] which lists needed packages.
+
  # go back to the webos-ports dir to continue with the build process
 +
  $ cd webos-ports
 +
</code>
  
===Create the directories===
+
{{#ifeq: {{{device_codename}}}|qemux86| To build the VirtualBox Appliance Image: |To build the luneos-dev-image for a device ({{lc:{{{device_codename}}}}}):}}
{{Commentlink
 
|section=create-the-directories-for-{{{devicename}}}
 
}}
 
  
You will need to set up some directories in your build environment.
+
<code>
 +
  $ MACHINE={{lc:{{{device_codename}}}}} {{#switch: {{{device_codename}}}|qemux86 = bitbake -k luneos-dev-emulator-appliance| Tenderloin = bb luneos-dev-image | RaspberryPi2 = bb luneos-dev-image | RaspberryPi3 = bb luneos-dev-image | bb luneos-dev-package }}
 +
</code>
  
To create them:
+
After the build completes, you will find your image in <build env>/tmp-glibc/deploy/images/{{lc:{{{device_codename}}}}}/
<code>$ mkdir -p ~/bin
 
$ mkdir -p ~/android/system</code>
 
  
===Install the <code>repo</code> command===
+
== Clean up the work directory  ==
{{Commentlink
+
If you run out of disk space during a build, run the following command from the <tt>webos-ports</tt> directory, then build again:
|section=install-the-repo-command-for-{{{devicename}}}
+
<code>
}}
+
  $ rm -rf tmp-glibc # If you're building a version before Black tie the folder is called ''tmp-'''e'''glibc''
 +
</code>
 +
This removes all build things but OE luckily has a copy of every build result in a cache folder called ''sstate-cache'' which you should keep to let OE quickly rebuild everything from the cache instead to do it from scratch (next build will be quite faster).
  
Enter the following to download the "repo" binary and make it executable (runnable):
+
If you run out of space because of a too large sstate-cache folder you can clean it up to contain only the results for the latest component versions with the following command:
<code>$ curl <nowiki>https://dl-ssl.google.com/dl/googlesource/git-repo/repo</nowiki> > ~/bin/repo
+
<code>
$ chmod a+x ~/bin/repo</code>
+
  $ openembedded-core/scripts/sstate-cache-management.sh -L --cache-dir=sstate-cache -d -y
 +
</code>
  
===Put the <code>~/bin</code> directory in your path of execution===
+
==Adding Swap Space==
Make sure that the <code>~/bin</code> directory you just created is in your [[Doc:_paths|path of execution]] so that you can easily run the <code>repo</code> command even when you're not in <code>~/bin</code>.  Assuming you are using the [[wikipedia:Bash_shell|BASH shell]], the default in recent versions of Ubuntu, you can set it like this:
+
To speed up the build, you can add extra 2GB of swap file like this:
<code>$ export PATH=${PATH}:~/bin</code>
 
{{tip|tip=You can make this change to the path permanent for all future Terminal sessions:
 
<code>$ gedit ~/.bashrc</code>
 
This will launch a graphical text editor.  Enter
 
<code>export PATH=${PATH}:~/bin</code>
 
on its own line, then save the file.}}
 
  
=== Initialize the CyanogenMod source repository ===
+
<code>
{{Commentlink
+
  $ dd if=/dev/zero of=swap_2gb.img bs=1024k count=2048
|section=initialize-source-repository-for-{{{devicename}}}
+
  $ mkswap swap_2gb.img
}}
+
  $ sudo swapon swap_2gb.img
 +
</code>
  
Enter the following to initialize the repository:
+
If you want it permanently add it to your /etc/fstab.  If the swap file is /var/swap_2gb.img, add the following lines to /etc/fstab:
 +
<pre>
 +
# additional swap
 +
/var/swap_2gb.img swap          swap    sw              0      0
 +
</pre>
  
<code>$ cd ~/android/system/
+
==Speeding up the build==
$ repo init -u <nowiki>git://github.com/CyanogenMod/android.git</nowiki> -b {{#if:{{{branch|}}}|{{{branch}}}|cm-10.1}}</code>
+
Bitbake automatically uses all of the processors assigned to your virtual machine and gives good result on a variety of systems.
 +
You can tune bitbake for modest improvements in build speed, but the correct parameters depend on whether builds are limited by processors, disk I/O, or network I/O.
 +
Edit webos-ports/conf/local.conf and uncomment the PARALLEL_MAKE and BB_NUMBER_THREADS lines.  
  
{{#if:{{{local_manifest|}}}|=== Add a local_manifest.xml file ===
+
Example values:
The {{{device}}} device requires special repositories that are currently not in the regular cyanogenmod source code.  To get these, repositories, you will need to create a file in ~/android/system/.repo/local_manifest.xml
+
<code>
 
+
  PARALLEL_MAKE = "-j 4"
and add the pasted contents of this page into it:  http://pastebin.com/FrFJmVtJ
+
  BB_NUMBER_THREADS = "4"
 
+
</code>
What is this file?  The <code>local_manifest.xml</code> file supplements the regular manifest, adding this needed repository to those provided by the main cyanogenmod ones so that a regular “<code>repo sync</code>” command will update changes from there as well.
 
 
 
Helpful tip!  If you’re using Ubuntu (or the GNOME desktop in general) you can create/edit this text file with the following command:
 
 
 
<code>$ gedit ~/android/system/.repo/local_manifest.xml</code>}}
 
 
 
=== Download the source code ===
 
{{Commentlink
 
|section=download-source-code-for-{{{devicename}}}
 
}}
 
 
 
To start the download of all the source code to your computer:
 
<code>$ repo sync</code>
 
 
 
The CM manifests include a sensible default configuration for <code>repo</code>, which we strongly suggest you use (i.e., don't add any options to ''<code>sync</code>'').  For reference, our current default values are <code>-j4</code>, and <code>-c</code>. The “<code>-j4</code>” part means that there will be four simultaneous threads/connections.  If you experience problems syncing, you can lower this to <code>-j3</code> or <code>-j2</code>. “<code>-c</code>” will ask repo to pull in '''only''' the current branch, instead of the entire CM history.
 
 
 
Prepare to wait a long time while the source code downloads.
 
 
 
{{Tip
 
|tip=The <code>repo sync</code> command is used to update the latest source code from CyanogenMod and Google. Remember it, as you can do it every few days to keep your code base fresh and up-to-date.}}
 
{{#ifeq:{{{branch|}}}|gingerbread|<!--donothing-->|
 
=== Get prebuilt apps ===
 
{{Commentlink
 
|section=get-prebuilt-apps-for-{{{devicename}}}
 
}}
 
Next,
 
 
 
<code>$ cd ~/android/system/vendor/cm</code>
 
 
 
then enter:
 
 
 
<code>$ ./get-prebuilts</code>
 
 
 
You won't see any confirmation- just another prompt.  But this should cause some prebuilt apps to be loaded and installed into the source code.  Once completed, this does not need to be done again.}}
 
{{#ifeq:{{{branch|}}}|gingerbread|
 
=== Get prebuilt Rom Manager ===
 
{{Commentlink
 
|section=get-prebuilt-rom-manager-for-{{{devicename}}}
 
}}Next,
 
 
 
<code>$ cd ~/android/system/vendor/cyanogen</code>
 
 
 
then enter:
 
 
 
<code>$ ./get-rommanager</code>
 
 
 
You won't see any confirmation- just another prompt.  But this should cause the Rom Manager apps to be loaded and installed into the source code.  Once completed, this does not need to be done again.}}
 
 
 
=== Prepare the device-specific code ===
 
{{Commentlink
 
|section=prepare-the-specific-code-for-{{{devicename}}}
 
}}
 
 
 
After the source downloads, ensure you are in the root of the source code (<code>cd ~/android/system</code>), then type:
 
 
 
{{#ifeq:{{{branch|}}}|gingerbread|
 
<pre&lt;noinclude&gt;&lt;/noinclude&gt; style="color:green">$ source build/envsetup.sh
 
$ lunch</pre>
 
 
 
You should see a list of devices, including something like <code>cm_{{{device}}}-userdebug</code>. Select it by typing its number. It is possible that <code>lunch</code> does not display your device. In that case try
 
 
 
<code> $ lunch cm_{{{device}}}-userdebug</code>
 
 
 
or
 
 
 
<code>$ lunch full_{{{device}}}-userdebug</code>
 
 
 
If all goes well, you should see that {{{device}}}-specific directories are downloaded automatically.|
 
<pre&lt;noinclude&gt;&lt;/noinclude&gt; style="color:green">$ source build/envsetup.sh
 
$ breakfast {{{device}}}</pre>
 
This will download the device specific configuration and kernel source for your device.
 
{{Note
 
|note=You '''MUST''' be using the newest version of '''repo''' or you will encounter errors with breakfast! Run <code>repo selfupdate</code> to update to the latest.
 
}}
 
}}
 
{{Tip
 
|tip= If you want to know more about what "<code>$ source build/envsetup.sh</code>" does or simply want to know more about the <code>breakfast</code>, <code>brunch</code> and <code>lunch</code> commands, you can head over to the [[Envsetup_help]] page}}
 
 
 
=== Extract proprietary blobs ===
 
{{Commentlink
 
|section=extract-proprietary-blobs-for-{{{devicename}}}
 
}}
 
Now ensure that your {{{devicename}}} is connected to your computer via the USB cable and that you are in the <code>~/android/system/device/{{{vendor}}}/{{{device}}}</code> directory (you can <code>cd ~/android/system/device/{{{vendor}}}/{{{device}}}</code> if necessary).  Then run the <code>extract-files.sh</code> script:
 
 
 
<code>$ ./extract-files.sh</code>
 
 
 
You should see the proprietary files (aka “blobs”) get pulled from the device and moved to the right place in the <code>vendor</code> directory. If you see errors about [[adb]] being unable to pull the files, [[adb]] may not be in the path of execution.  If this is the case, see the [[adb]] page for suggestions for dealing with "command not found" errors.
 
 
 
{{Note
 
|note=Your device should '''already be running''' the branch of CyanogenMod you wish to build your own version of for the extract-files.sh script to function properly. If you are savvy enough to pull the files yourself off the device by examining the script, you may do that as well without flashing CyanogenMod first.
 
}}
 
{{Note
 
|note=It’s important that these proprietary files are properly extracted and moved to the <code>vendor</code> directory. Without them, CyanogenMod will build without error, but you’ll be missing important functionality, such as the ability to see anything!'''
 
}}
 
 
 
 
 
=== Turn on caching to speed up build ===
 
{{Commentlink
 
|section=turning-on-caching-for-{{{devicename}}}
 
}}
 
 
 
If you want to speed up subsequent builds after this one, type:
 
 
 
<code>$ export USE_CCACHE=1</code>
 
 
 
{{Tip
 
|tip=Instead of typing <code>cd ~/android/system</code> every time you want to return back to the root of the source code, here’s a short command that will do it for you: <code>croot</code> . To use this command, as with <code>brunch</code>, you must first do “<code>. build/envsetup.sh</code>” from <code>~/android/system</code>. Notice there is a period and space (“<code>. </code>”) in that command.
 
}}
 
 
 
 
 
=== Start the build ===
 
{{Commentlink
 
|section=start-the-build-for-{{{devicename}}}
 
}}
 
 
 
Time to start building! So now type:
 
 
 
<code>$ croot
 
$ brunch {{{device}}}</code;>
 
 
 
The build should begin.
 
 
 
{{Tip|tip=If the build doesn't start, try <code>lunch</code> and choose your device from the menu.  If ''that'' doesn't work, try <code>breakfast</code> and choose from the menu.  The command <code>make {{{device}}}</code> should then work.}}
 
 
 
{{Tip|tip=A second, bonus tip!  If you get a '''command not found''' error for <code>croot</code> or <code>brunch </code> or <code>lunch</code>, be sure you’ve done the “ <code>. build/envsetup.sh</code>” command in this Terminal session from the <code>~/android/system</code> directory.
 
}}
 
 
 
===If the build breaks...===
 
{{Commentlink
 
|section=if-the-build-breaks-for-{{{devicename}}}
 
}}
 
 
 
* If you experience this not-enough-memory-related error...
 
 
 
<code>ERROR: signapk.jar failed: return code 1make<nowiki>: *** [</nowiki>out/target/product/{{{device}}}/cm_{{{device}}}-ota-eng.root.zip] Error 1</code>
 
 
 
...you may want to make the following change to:
 
 
 
<code>$ system/build/tools/releasetools/common.py</code>
 
 
 
Change: <code>java -Xmx2048m</code> to <code>java -Xmx1024m</code> or <code>java -Xmx512m</code>
 
 
 
Then start the build again (with brunch).
 
 
 
* If you see a message about things suddenly being “killed” for no reason, your (virtual) machine may have run out of memory or storage space. Assign it more resources and try again.
 

Latest revision as of 21:49, 30 September 2017

Setup the build environment

Before you can build, you will need some tools. If you try to build without them, bitbake will fail a sanity check and tell you about what's missing, but not really how to get the missing pieces.

If your distribution is Ubuntu, you need to be running the bash shell. You can be sure you are running this shell by entering the following command and selecting "No" at the prompt:

    $ sudo dpkg-reconfigure dash
          

The essential and graphical support packages you need for a supported Ubuntu or Debian distribution are shown in the following commands:

    $ curl -sL https://deb.nodesource.com/setup_4.x | sudo bash -
    $ sudo apt-get install gawk wget git-core diffstat unzip texinfo \
    build-essential chrpath libsdl1.2-dev xterm nodejs bzr

(Recently, building on 64-bit Ubuntu 16.04 also required apt-get install gcc-multilib g++-multilib.)

For other Linux distributions please have a look at the Yocto Project Quick Start guide.

Note:

The steps below will setup your build environment for a development build. If you want to build the latest release use ENV_NAME=stable instead but keep in mind that all new developed features need to go into testing first before they are submitted to stable.

 $ cd into-your-build-directory
 $ mkdir webos-ports-env && cd webos-ports-env
 $ export ENV_NAME=testing
 $ wget https://raw.github.com/webOS-ports/webos-ports-setup/$ENV_NAME/Makefile
 $ make setup-webos-ports

This has been tested on Gentoo (shr-chroot) and Ubuntu-12.04/13.04/14.04/15.10/16.04 amd64/x64 and should work almost everywhere where valid toolchain is provided.

You need a lot of RAM to link QtWebEngine, make sure you have at least 6GB (it's OK when some of that is swap, because it's used only for short part of build). QtWebEngine needs so much ram to link because it's linking with debug symbols (huge files) which are stripped later in do_package after creating -dbg packages.

Building

Make sure you're _not_ root, as the build will fail. If you SU in the middle, don't forget to set up env afresh (. setup-env)

To configure to build (notice '.' which is actually bash 'source' command):

 $ cd into-your-build-directory/webos-ports-env/webos-ports
 $ . setup-env

To update metadata

 # go one level up to the webos-ports-env dir
 $ cd ..
 $ make update
 # or if it shows warning about different bblayers.conf or layers.txt
 $ make update-conffiles && make update
 # you can also add UPDATE_CONFFILES_ENABLED = 1 to config.mk
 # if you never want to have any uncommited changes in your checkouts RESET_ENABLED = 1 in config.mk
 # go back to the webos-ports dir to continue with the build process
 $ cd webos-ports

To build the luneos-dev-image for a device ({{{device_codename}}}):

 $ MACHINE={{{device_codename}}} bb luneos-dev-package

After the build completes, you will find your image in <build env>/tmp-glibc/deploy/images/{{{device_codename}}}/

Clean up the work directory

If you run out of disk space during a build, run the following command from the webos-ports directory, then build again:

 $ rm -rf tmp-glibc # If you're building a version before Black tie the folder is called tmp-eglibc

This removes all build things but OE luckily has a copy of every build result in a cache folder called sstate-cache which you should keep to let OE quickly rebuild everything from the cache instead to do it from scratch (next build will be quite faster).

If you run out of space because of a too large sstate-cache folder you can clean it up to contain only the results for the latest component versions with the following command:

 $ openembedded-core/scripts/sstate-cache-management.sh -L --cache-dir=sstate-cache -d -y

Adding Swap Space

To speed up the build, you can add extra 2GB of swap file like this:

 $ dd if=/dev/zero of=swap_2gb.img bs=1024k count=2048
 $ mkswap swap_2gb.img
 $ sudo swapon swap_2gb.img

If you want it permanently add it to your /etc/fstab. If the swap file is /var/swap_2gb.img, add the following lines to /etc/fstab:

# additional swap
/var/swap_2gb.img swap          swap    sw              0       0

Speeding up the build

Bitbake automatically uses all of the processors assigned to your virtual machine and gives good result on a variety of systems. You can tune bitbake for modest improvements in build speed, but the correct parameters depend on whether builds are limited by processors, disk I/O, or network I/O. Edit webos-ports/conf/local.conf and uncomment the PARALLEL_MAKE and BB_NUMBER_THREADS lines.

Example values:

 PARALLEL_MAKE = "-j 4"
 BB_NUMBER_THREADS = "4"