From Mandriva Community Wiki

Contents 1 History 2 One run of mkcd

The very generic goal of mkcd is to put packages and files into an ISO 9660 image. Of course, it does a bit more than that otherwise more than 8000 lines of perl would have been a bit overkill.

[edit] History

In late 1999 I took charge of building the Mandrakelinux ISO images (by that time it was name Linux-Mandrake). We then had only one disc for the whole distribution, and the main trouble was to make sure that all the dependencies were verified, and the ISO made correctly. There was a collection of several scripts doing simple tasks (this is what UNIX is about).

With the Linux-Mandrake 7.1, we decided to create a 2 CD installation, then the need to choose were to put packages appeared. The various scripts making the mkcd collection improved to help selecting which packages to include and on which CD to put them.

Which packages to include was quite easy as nearly everything was included, except what didn't work.

The work to select on which CD to put a package, however, was more complex, as you need to be sure all the dependencies for that package are already honored to be able to include it. A particularity of the way installation deals with media is that it cannot install packages foo_A from CD 1 when one of its dependencies is present on CD 2 or more. As a consequence the mkcd collection started to become more complex to check that any packages put on the CD have its dependencies correctly inserted.

By that time the package selection was done with lists manually updated, including all the dependencies. The scripts were able to tell you when a package was missing, but not able to find it and automatically insert it. I have done important work creating the version 2 of mkcd which was able to do such a job. Then the lists were simpler as only the important packages were listed and all the dependencies automatically inserted.

That was the time when I consider mkcd became really useful. Then the Mandrakelinux product lines became more complex and there appeared various CD combinations with some shared CD.

During the years 2003 and 2004, I added the support for an update CD (a new installation CD able to use old CD's and just bring in some new packages), for cloning a CD (copy one existing CD except some files), better dependency checking (now the algorythm to compute dependencies is specific to mkcd), the ability to build a CD without using disk space (only doing it from memory and one particular feature of mkisofs)...

[edit] One run of mkcd

Let's look at one run of mkcd.

mkcd first reads the config files (in Tools.pm function config) given with the -s option. The syntax of these config files is checked and incoherency or strangeness are reported with some warnings (parsed with the pCommandline.pm function parseCommandLine). In this part, it constructs a global structure which contains the various disc layouts and more specifically generates the list of RPMs used by the discs.

Then mkcd checks the -m option which gives the CD to build. From that list it creates the 'groups' which will become the base for the disc generation (done in Group.pm makeWithGroups function, called from the mkcd make function). A group is a set of directories related to the same installation procedure. The groups are calculated into the function getGroups in Group.pm.

list 1 -k pieces/10.1/i586/media/media_info/pubkey_main mkcd/10.1/tools mkcd/10.1/main
rpmlist -b pieces/10.1/extra/RPMS -s pieces/10.1/extra/SRPMS
rpmlist -b pieces/10.1/i586/media/main -s pieces/10.1/SRPMS/main

list 2 -k pieces/10.1/i586/media/media_info/pubkey_contrib mkcd/10.1/contrib
rpmlist -b pieces/10.1/i586/media/contrib -s pieces/10.1/SRPMS/contrib

#
disc 1 650m 101-20040715-1 "Installation CD 1" "Mandrakelinux-Cooker-Disc1" -p Mandrakesoft
dir rpms media/main
generic --synthesis rpms 1
advertising pieces/10.1/ads/dwd-01.png \
pieces/10.1/ads/dwd-02.png
installation -o 50,3,0.1 --synthesis -l fr,it,en,de,es -t download-i586 -i pieces/10.1/install/i586/ -d 1/rpms 2/rpms 3/rpms

disc 2 650m 101-20040715-1 "Installation CD 2 (x86)" "Mandrakelinux-Cooker-Disc2" -p Mandrakesoft
dir rpms media/main2
generic --synthesis rpms 1

disc 3 650m 101-20040715-1 "Installation CD 3 (x86)" "Mandrakelinux-Cooker-Disc3" -p Mandrakesoft
dir rpms media/main3
generic --synthesis rpms 1

In this example, if the command line is mkcd -s file.spec -m 1-3, then mkcd which checks the disc 1 and sees that the installation command on this disc is using the packages from the 1/rpms, 2/rpms and 3/rpms, 1/rpms are packages from the list 1 of the /media/main directory of disc 1, 2/rpms on disc 2 and 3/rpms on disc 3.

As a consequence, a group for these 3 directories will be created, the master disc will be the disc 1, and the disc 2 and 3 will depend upon it.

Then mkcd will order the disc, quite simple in this example, and will create the discs in the order 3, 2 then 1. This is done with the orderGroup function of Group.pm.

Once the group is created and the disc order resolved, mkcd tries to separate the various groups into 'IOgroups'. An 'IOgroup' is a set of groups with no relations between each other. In the following example, disc 1 and 2 are totally independant:

list 1 -k pieces/10.1/i586/media/media_info/pubkey_main mkcd/10.1/tools mkcd/10.1/main
rpmlist -b pieces/10.1/extra/RPMS -s pieces/10.1/extra/SRPMS
rpmlist -b pieces/10.1/i586/media/main -s pieces/10.1/SRPMS/main

list 2 -k pieces/10.1/i586/media/media_info/pubkey_contrib mkcd/10.1/contrib
rpmlist -b pieces/10.1/i586/media/contrib -s pieces/10.1/SRPMS/contrib

#
disc 1 650m 101-20040715-1 "Installation 1 CD 1" "Mandrakelinux-Cooker-Disc1" -p Mandrakesoft
dir rpms media/main
generic --synthesis rpms 1
advertising pieces/10.1/ads/dwd-01.png \
pieces/10.1/ads/dwd-02.png
installation -o 50,3,0.1 --synthesis -l fr,it,en,de,es -t download-i586 -i pieces/10.1/install/i586/ -d 1/rpms

disc 1 4.3g 101-20040715-1 "Installation 1 DVD" "Mandrakelinux-Cooker-Disc1" -p Mandrakesoft
dir rpms media/main
generic --synthesis rpms 1
advertising pieces/10.1/ads/dwd-01.png \
pieces/10.1/ads/dwd-02.png
installation -o 50,3,0.1 --synthesis -l fr,it,en,de,es -t download-i586 -i pieces/10.1/install/i586/ -d 2/rpms

Once those IOgroups have been created, then mkcd first creates the 'static' part of the disc. This includes directories, the installation files, and any files which are hard copied to the disc structure. In this run mkcd calculates the remaining space available and, more specifically, it will maintain an up to date disc size and space left for any operation performed. During the whole process of disc creation, mkcd will not copy or move any package files on the hard disks, everything will be done in memory and only when the disc is considered finished will the files be practically generated and the ISO files created.

Regarding space saving, mkcd tries to use hard links to save disk space, and, moreover, you can use the --nolive option to force mkcd to calculate everything only in memory (it will still generate some files on the harddrive, though, but only some tens of megabytes for all the discs).

mkcd then checks the potentially already built CD's (function getBuiltDiscs from Disc.pm) and includes their content into its table (you may want to create a new disc 1 and keep the already built disc 2 and disc 3).

Then the real package sorting job starts. All the RPM packages are listed (in getGroupReps from Group.pm) and then parsed (in genDeps from Packages.pm) to generate a hdlist as well as a perl-URPM structure to handle dependencies.

All the package sizes are checked by the getSize function of Package.pm.

A heuristic tries to guess the future hdlist size (function guessHdlistSize of Package.pm) to reserve the necessary space on the installation disc.

The rpmsrate file is then read and parsed, to get all the sections and the various levels associated with packages (cleanrpmsrate function of Tools.pm).