rosa-abf

[rosa-abf]

Automatic Build Restart

From time to time ABF users may notice that package build lasts significantly longer than usual and it looks like it was restarted in the middle. If you meet such a case, don't panic - in some cases this is expected behavior. In particular, if some of the repositories used to install build dependencies was changed in the middle of dependency installation, it is possible that package manager will fail to find some package and installation process will be aborted. 

ABF is able to recognize such situations by parsing build logs. If such a case is detected, then ABF will wait a little (to allow all new packages to be published) and will restart the build from scratch. In the logs, this will be remarked by the following messages:

Repository was changed in the middle, will rerun the build

or

Repository was changed in the middle, will rerun the tests

or something like this.

Number of restart attempts is limited to 5, so the build will not loop forever even if repository metadata is completely broken.

The automatic restart feature was introduced several months ago and according to our statistics, thousands of builds were silently restarted from that time and a lot of maintainers’ time was saved, since otherwise they would have to look at the logs and simply restart the build after quick log analysis.

Note that detection of the fact that the build failed due to repository changes is not perfect. The build is restarted if build log contains one of the lines mentioned in the RETRY_GREP_STR variable in build scripts.

Currently it looks like the following:

RETRY_GREP_STR="You may need to update your urpmi database\|problem reading synthesis file of medium\|retrieving failed: "

If you know more patterns that should be added to that set - let us know or just send pull request.

[rosa-abf]

Console Client: Smarter "abf put"

Console client automates a lot of tasks, but sometimes it turns out to be not smart enough and does the things that you don't want it to do.

In particular, from time to time users put some extra files in a folder with a cloned Git repository of their ABF projects. Common examples include:

folders with unpacked tarballs with upstream source code

rpm files created by "abf rpmbuild"

other binary files that for some reason were copied to the current folder but were not uploaded to ABF File Store since the console client detected that these files are not used in the spec

Up to now, "abf put -m <msg>" command would add all such files and folders to the Git, but in almost all cases this is not what users expect. The thing is that the console client is used to control packaging projects that normally consist of spec file for rpmbuild, some patches and other textual files. However, none of these projects contain folders. As for binary files, then according to ABF conventions they should be all stored in a File Store, not in Git.

That's why it is not wise for the console client to add folders and binary files to Git. And from now on, it doesn't do this - at least not by default. To be sure, you still can add folders and binary files by means of corresponding options:

$ abf put -m "I really want these files" --add-folders --add-binaries

But we believe that you will never need these functionality. Instead, the new default behavior will save you some time since you won't have to clean Git repository from garbage files.

[rosa-abf]

Activity Feed Improvements

Good new for those who use ABF activity feeds for monitoring but lack filtering possibilities - we have added filters by project owner and project name, so now you can easily limit the projects being displayed. In addition, we have added a new tab named “Own activity” which display your own activity in ABF.

Hopefully this will help you not to get lost among thousands of tasks performed in ABF every day.

[rosa-abf]

OpenMandriva Development Process

A small promo clip that demonstrates basic possibilities of ABF Web interface - let’s build a package for OpenMandriva!

[rosa-abf]

Android Client

From the very first ABF days our users experienced interest in a client for mobile devices. Though ABF Web UI is accessible from any browser, it is not very convinient to work with it from smartphones or tablets. With the last update, we have tried to make Web UI more suitable for mobile users, but on many devices its usability is still far from perfect. In addition, Web UI is not always convenient even for desktop users - that's why we have several console clients that are actively used by maintainers.

On the other hand, there are not so many actions that can be performed from mobile device - for example, it is unlikely that you will write and commit a lot of code since you are not able to try and debug it when working on smartphone. In fact, number of mobile visitors at ABF is very small so we didn't bother much about mobile clients up to now. However, the ROSA Company has a wonderful opportunity to attract student of Russian Higher School of Economics (HSE) to different projects and this year a student team from HSE has developed the ABF Android Client for us.

The latest version of the client (apk file) can be downloaded from ABF File Store. The source code is located in ABF, as well. .abf.yml file in that project always contains a hash sum of the latest apk file. You can use that hash to download the apk file from the File Store. Currently we have to follow this way to distribute our Android client, since currently ABF doesn't have any means to build Android applications.

Currently the client currently requires modern Android 4.x to work. If you want to add support for earlier versions, feel free to download the source code and try to compile it in Android Studio for your device. If you succeed, let us know (and feel free to send patches if you have to make any changes).

Once that APK file is downloaded and installed, you will be able to launch ABF Client application. After signing in, you will be able do checkout projects from ABF. For every project, you can observer its files, history of commits to Git repository and status of its builds at ABF. You can edit text files with a simply embedded editor or with any external program on your device. The client allows uploading binary files to file store (hash sums are automatically added to .abf.yml) and downloading files specified in .abf.yml.

The client provides rich possibilities to work with Git. And finally, you can launch new builds for your projects on ABF directly from the client:

Feel free to download and try the client. Unfortunately, we don't have much resources to work hard on the client improvements, but we are open to any suggestions, improvements and fixes. Patches are welcome, as well as APK files for different devices that are not supported at the moment.

[rosa-abf]

Updated Task Monitoring Pages

After releasing the new ABF Web UI, we have received rich feedback and a lot of useful suggestions. Most of them are minor ones, but there are some significant improvements which should make ABF Web UI even more useful. One of such improvements is redesign of Task Monitoring pages.

In the new design, we have moved the Filters tab at the left side of the page. Note that this happens only if you monitor is large enough; on small monitors or in small windows the tab will be automatically moved to its previous location, above the table with the build lists. In addition, we have moved New Build button (available on task monitoring pages of every project if you have permission to build that project) out of the Filters tab, so now it can be accessed more easier.

Finally, we have removed the separate section with status of build servers and tasks. Now this information is simply displayed at the top of the page, under the main menu.

Hopefully this updates will make your work a little more efficient.

[rosa-abf]

Adding Remote Git With abfcc

In ABF, you can often find a lot of projects with the same name that belong to different groups/users. This projects use different Git repositories, but from time to time users want to migrate changes from one project to another. One of the possible ways to do this is to send a pull request, but this is mostly used in cases when you don't have permissions to write to the target project or want somebody to take an additional look at your changes.

If you have all necessary permissions and just want to merge changes from one project to another, you can use git remote command to add remote repository, git fetch to fetch its content and then git merge to merge changes from remote repository to the current one. For example, let's merge rosa2014.1 branch of import/foo project into the master branch of openmandriva/foo:

$ abf get openmandriva/foo && cd foo $ git remote add import https://<username>@abf.io/import/foo.git $ git fetch import $ git merge import/rosa2014.1

One may notice that you have to type too many letters to perform this operation. To help you a bit, we have implemented remote command in http://wiki.rosalab.ru/en/index.php/ABF_Console_Client#remote. With this command, the same task requires less efforts:

$ abf get openmandriva/foo && cd foo $ abf remote import $ git merge import/rosa2014.1

As you can see, abfcc automatically adds a remote repository of a project with the same name from the import group. Remote repository is named "import", as well.

You can fetch a repository of a project with different name - just specify project name after the group name. For example, this command will add remote repository of import/foo2.

$ abf remote import foo2

ABF console client constantly gets minor improvements, so check its changelog from time to time. And feel free to propose your suggestions and patches!

[rosa-abf]

"Build For Platform" Customization

When you launch a new build, you should choose the platform to build for. From the beginning of ABF, the list of possible platforms included all platforms for which you had access to - in particular, all public platforms. As ABF grows, the number of platforms increases and now it is not hard to get lost in their list. But from now on, this problem can be easily eliminated.

First of all, if on the "New Build" page you choose a repository which is connected to a particular platform, then only this platform will be displayed in the "Build for platform" column. For example, if we select "rosa2014.1/main" repository, then ABF will only display "rosa2014.1" platofrm. This not only simplifies the selection, but also guarantees that you can't accidentally build a package for rosa2014.1 repositories using repositories of some other platforms.

If the repository is not connected to any platforms (e.g., you choose a personal repository), then this limitation doesn't work. However, now you can customizae the list of platforms displayed in this case in your user settings. Just go to the "Build settings" tab and mark the platforms you want to be displayed when you creates a new build list and target platform can't be guessed automatically. Note that if you don't select any platform in these settings, they will be ignored and all available platforms will be displayed.

[rosa-abf]

New Design!

More than two months passed after the last new in this blog, but this doesn't matter that ABF developers are lazy or have very long vacations on New Year holidays. During the last several months, we worked hard on ABF code refactoring and now we are ready to present the results of our work.

At the backend side, we have migrated to Bootstrap 3 and AngularJS frameworks.This allowed to get rid of large pieces of code that became hard to maintain, so we hope that in future we will be able to implement the new features faster.

As a result of this switch to new frameworks, ABF UI has got a lot of changes. Just look at the new screenshots:

* Build lists:

* Project list:

* Statistics:

... or simply got to http://abf.io and try it!

Remember that the changes are much deeper than just UI and the size of the changes is huge (the diff between old and new implementation is more than 20Mb). We have tested our new developments as much as we could, but it is still possible that we have missed some issued. So please be patient and report us any problems you meet, we will try to fix them as soon as we can.

[rosa-abf]

Build Chains In ABF Console Client

It often happens that projects have long chains of dependent packages and if you update one package, you have to rebuild a lot of others in a certain order. That is, you should build a package, wait for it to be published (or for container to be created), then launch build of the next package and so on. Up to now, the only way to automate this task was to use dependent mass builds in ABF Web UI, but this way is not very convenient, especially if you have long chains with short links.

To solve this problem, we have implemented new "chain_build" option in ABF Console Client. The usage is very easy, just look at the example:

 abf chain_build proj1 proj2:proj3 proj4

Here we will first launch build of proj1, then builds of proj2 and proj3 will be launched in parallel and only after that build of proj4 will be initiated. If any of the builds fails, the whole chain is aborted and successive builds are not launched. If you specify "--auto-publish" option, the next link of the chain will be processed only after all builds from the previous link are published. Alternatively, you can specify "--auto-create-container" option. In this case, console client will wait until all containers for the link builds are created and then attach all these containers to all successive builds.

Note that unlike "abf build" command, chain_build doesn't automatically detect values for different options. You have to manually specify all options necessary for the build - in particular, "save-to" repository and repositories to build with. So a real-life example looks like the following:

abf chain_build urpmi apache:dracut texinfo -b rosa2014.1 -s abf_personal/main -r rosa2014.1/main -r rosa2014.1/contrib --auto-publish-status=none --auto-create-container

More details are available at ROSA Wiki - http://wiki.rosalab.ru/en/index.php/ABF_Console_Client#chain_build

To implement the new "chain_build" possibility, we had to add additional small features to already existing functionality. A couple of them can be interesting for ABF Console Client users:

abf status now displays status of container and container path, if any

abf build now accepts additional "--build-list" option to attach a given build list to the new build. This option can be specified more than once.

Updated ABF Console Client is already available in ROSA Desktop Fresh R4 and OpenMandriva Cooker repositories. Update for ROSA Server is coming soon.

[rosa-abf]

ABF Online Statistics

New users and third-party visitors often ask questions about activity of maintainers and developers in ABF - how many packages they build, how many commits make and so on.

To be sure, one can monitor all tasks using Task Monitoring, but it can't reflect all that events that happen inside ABF. To provide users with more complete and convenient statistics, we have implemented a separate page with corresponding graphs:

Here you can see statistics of all builds, Git commits, pull requests and project issues for any given period of time. Feel free to estimate how hard ABF is utilized.

[rosa-abf]

ABF Console Client 2.0

The major change in this release is that the console client now has localization facilities and is provided with complete Russian localization.

If you want to add localization for other languages, feel free to take pot file from our Git repository and prepare a po file for your language. Currently we don't use any localization service, but we can consider using one if requested.

From the functionality updates, one should mention new "update" command which can be used to modify settings of a given project. For example, you can turn on publish_i686_into_x86_64 feature using the following command:

 abf update -p server/httpd --biarch=true

Please launch abf update --help to get a list of supported options.

The build task got some new options allowing to set additional build parameters:

--no-extra-tests - do not launch extra comprehensive tests

--testing - include "testing" subrepository

--cached-chroot - use cached chroot for the build

--save-chroot - save build chroot in case of failure

Finally, abf put now accepts --upload-all option to upload all binary files from the current folders to File Store, without checking which of them are really used in the spec file. This feature is now automatically used when creating a project from SRPM. So from now on you are able to create a project for a system which differs from your current one and whose RPM spec files cannot be parsed by your rpm.

ABF Console Client 2.0 is already available in repositories of ROSA Enterprise Linux Server, ROSA Desktop Fresh R4, OpenMandriva 2014.1 and Cooker.

[rosa-abf]

Building RPMs Directly From ABF Git

This time we won't discuss any new features. Instead, we will give you a hint about increasing your productivity by using existing ABF possibilities.

As far as you know, we encourage users not to store binary files in ABF Git. Instead, such files should go to the File Store, and Git should only contain .abf.yml file with references to them. The most common example of such binary file is a tarball with source code used to build a package. Usually such tarballs are downloaded from upstream and stored in ABF File Store in order to guarantee that they can be accessed at any time. But what if upstream itself is located in ABF?

For such projects, one can download source code archives corresponding to any Git branch or tag. For example, tarball with source code of soft/abf-console-client project corresponding to a tag named "v1.99" can be accessed by the following URL:

https://abf.io/soft/abf-console-client/archive/abf-console-client-v1.99.tar.gz

If build tools that you use to build packages are able to download remote files, then you can specify such URLs as code sources. For example, rpmbuild in ROSA Desktop Fresh works fine with external sources. So to build your ABF project for ROSA Desktop Fresh, there is no need to create a tarball and upload it to File Store - you can simply specify tarball URL and use tags to indicate versions. For example, take a look at how this is implemented for ROSA Freeze:

Here we download source tarball from https://abf.io/soft/rosa-freeze/, assuming that the project contains a tag corresponding to the requested version. When a new version is released, a corresponding tag is set at upstream project, and you should just bump version in rosa-freeze.spec to build a package that new version.

So save your time, develop, keep your code and build your projects in ABF!

[rosa-abf]

Saving Build Chroot in ABF

Your build fails in ABF and you can't find out why? Not a big problem anymore - from now on, in case of failure you can get the whole chroot used to build your package. Just set a corresponding option when starting a build.

If your build fails, then among the logs you will be able to find rpm-buildroot.tar.gz file. Feel free  to download it, unpack and chroot into it. Rpmbuild folders are located inside builddir/build folder.

[rosa-abf]

New Build Status - "Unpermitted Architecture"

It sometimes happens when application can;t be built for all hardware architectures supported by ABF - for example, if developers use platform-specific instructions or just didn't have a chance to test their product on all machines. The latter is a common case for build failures on ARM platforms supported by ABF among others. For example, OpenMandriva maintainers build their packages for 32bit and 64bit ARM variations and quite a few programs fail to compile or to run successfully.

For RPM packages, there is a way to explicitly specify "forbidden" or "allowed" architectures - one should use ExcludeArch or ExclusiveArch tags in spec file correspondingly. If you try to build a package on a platform which is specified in the ExludeArch tag or your spec has ExlusiveArch tag define and the target platform is not included there, then rpmbuild will issue an error message: "architecture not included".

Up to now, ABF didn't provide a way to distinguish such errors from other rpmbuild failures - in any case, the build task in ABF would finish with Build Failed status. And it was a little annoying to analyze build results of mass builds where many packages failed due to unsupported architecture.

To simplify such analysis, we have added a special build status - "Unpermitted Architecture". From now on, for all MDV-platforms ABF analyzes ExlusiveArch and ExcludeArch tags after building SRPM package and declines to build a binary RPM if the target architecture is not supported. This makes build results more clear and at the same time saves several seconds for such builds, since ABF doesn't even try to build binary package.

[rosa-abf]

Task Monitoring And Mass Builds

One of the strongest ABF features often used by distribution developers is possibility to launch mass builds - simultaneous rebuild of a given set of packages or even of the whole repository.

A mass build is launched by a couple of mouse clicks, and you can monitor build process in the Task Monitoring tab, as usual. However, there was   a small inconvenience for a person who launched mass build - his monitoring statistics was really polluted by these builds and it was really hard to monitor builds launched manually (that is, not as part of some mass build).

A good news is that we have fixed this issue and improved filtering in the task monitoring - now you can simply choose "None" string in the "Mass Build" dropbox to hide all mass build tasks. So if you launch a mass build, there is no need to be afraid of polluting monitoring statistics anymore.

[rosa-abf]

ABF ER-Diagram - A Little Help for Novices

New users of ABF sometimes get confused by the number of different entities that exist in ABF. To simplify hteir life, we have prepared a diagram with the major ABF entities with some comments - check ROSA wiki for more details.