1040319 - Ensure that Fennec builds from mozilla-esr31 have a buildID to allow for armv6/Android 2.2 users to update to mozilla-esr31 apks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Lukas Blakk [:lsblakk] use ?needinfo]

This is the tracking bug for the results of today's meeting about EOL strategy for Android 2.2 & armv6.

"Effective next week we will take uplifts up to FF32 (beta) that will turn off armv6 and block installation of Firefox for Android to Android 2.2 users.  This will take effect on Nightly tomorrow and then Aurora/Beta after uplift.
 
For releases over the next 6 months there will be builds of armv6 (supporting Android 2.2) off the mozilla-esr31 branch, but this is not an ESR.  Matt Wobensmith who does the security QA for Firefox ESR will have no additional work specific to mobile releases from this branch.  Mobile QA will maintain a comparable level of effort on releases, with two builds (x86, armv7) from mozilla-release and then one (armv6) from mozilla-esr31.

To get ready for the first of these releases, in 6 weeks time when we ship FF32, RelEng has an action on them to confirm that they can build from the two repos with two separate requests for builds and get the buildID set correctly on them to ensure updates will be offered to the Android 2.2 users from the Play Store.  QA will also have an action item to confirm this works and that users do not need to uninstall/reinstall to be on the security-fix-only builds."

To resolve this bug fixed please confirm the buildID generation still works for the three apks as it does now when using a single repo.  QA needs to sign off that updates from FF31 GA to FF31.1 from mozilla-esr31 works.

Finally, flagging for relnote-firefox? as this will be one of the channels through which we can announce the de-support and that security-fix-only builds will end on January 6th.

Comment 1

[Aki Sasaki (not active)]

armv6 needs to be the lowest version code.
I was brainstorming that if ESR31 had code to subtract, say, 1 week from the build date, we could avoid tricky go-to-build timing issues.  Otherwise, if we kept the subtract-one-{hour?} from the build date, we'd have to make sure the esr build started in the same hour or before the release build.

Comment 2

[Aki Sasaki (not active)]

[13:33]	<aki>	coop: http://mxr.mozilla.org/mozilla-beta/source/mobile/android/base/Makefile.in#17
[13:33]	<aki>	specifically http://mxr.mozilla.org/mozilla-beta/source/mobile/android/base/Makefile.in#22
[13:33]	<coop>	aki: thanks
[13:33]	<aki>	should be a one liner
[13:35]	<aki>	make sure he knows it doesn't have to be a valid datetime string; subtracting 7*24 (or whatever units it's in) is valid, even if that means it goes to day 97 of the previous month
[13:35]	<aki>	just numbers that need to be greater than or less than
[13:36]	<catlee>	I thought I put that in the bug?
[13:36]	<aki>	dunno, i haven't read the bug :)
[13:36]	<aki>	https://bugzilla.mozilla.org/show_bug.cgi?id=1040319

Comment 3

[Aki Sasaki (not active)]

http://hg.mozilla.org/releases/mozilla-esr31/file/1e303589a899/mobile/android/base/Makefile.in#l22 since I think mxr's links aren't permanent (adding or subtracting lines at the beginning of the file will make the line link point to the wrong line).

Comment 4

[Aki Sasaki (not active)]

(In reply to Aki Sasaki [:aki] from comment #2)
> [13:35]	<aki>	make sure he knows it doesn't have to be a valid datetime
> string; subtracting 7*24 (or whatever units it's in) is valid, even if that
> means it goes to day 97 of the previous month
> [13:35]	<aki>	just numbers that need to be greater than or less than

I thought more about this, and this might suck if build day is the last of the month, and the ESR gets built a day later, on the first of the month. =\  Trying to figure out a way other than 1) convert to epoch time 2) subtract the correct # of seconds 3) re-convert to datetime of the appropriate format, but that may be the cleanest solution.

Comment 5

[Lawrence Mandel [:lmandel] (use needinfo)]

Pete - Ben tells me that you've picked up this work. Do you have a plan for testing this on beta before we hit release? What is your time line for completing this work?

Comment 6

[Pete Moore [:pmoore][:pete]]

Hi Lawrence,

That's right - I'm working on this over in bug 1042128 which I guess we can mark as a duplicate (or this a duplicate of that). In any case, I can briefly summarize my proposal here:

The versionCode in the apk meta data (which is derived from the buildid) needs to have a lower value for armv6 than the other architectures. It is treated as a plain integer. My proposal is to change the mechanism to generate the versionCode from a starting point of e.g. 2008080001, and then bump it by 1 with each release. This guarantees that:
a) the version number is higher than the previous releases for armv6 releases
b) it continues to increase
c) it will always take less precedence than the versionCode of the apk's for the armv7 and x86 apks.

If you'd like to review the full content of bug 1042128 (which gives a bit more detail) and let me know your feedback on this proposal, I'd be very happy to hear it. I asked Nick, Ben, Coop and Catlee also for their feedback, and I think we're all aligned currently that this proposal should work - but maybe you spot something we haven't. :-)

Many thanks,
Pete

Comment 8

[Pete Moore [:pmoore][:pete]]

For transparency, attaching my work so far, as it is end-of-day now and maybe someone wants to see it before I am back at work in the morning...

Comment 9

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_buildbot-configs_v1.patch

Comment 10

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_esr31_branch_v1.patch

Comment 11

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_tools_v1.patch

Comment 12

[Pete Moore [:pmoore][:pete]]

I guess I will also need a similar patch for betas? Would that be release-fennec-mozilla-beta.py.template?

I guess both betas and releases will get made off the esr31 branch(?) Hmmmm... What do you think, Rail? Not quite sure how that works.

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8475998 [details] [diff] [review]
bug1040319_buildbot-configs_v1.patch

Review of attachment 8475998 [details] [diff] [review]:
-----------------------------------------------------------------

Since this "hack will be going away when stop support for armv6, I'm OK to have a function in a config file to avoid pollution of libraries and scripts.

::: mozilla/release-firefox-mozilla-esr31.py.template
@@ +42,5 @@
>  releaseConfig['nextMilestone']       = releaseConfig['nextAppVersion']
> +
> +def bumpIntegerInFile(previousContents):
> +    print previousContents[0]
> +    return str(int(previousContents[0]) + 1)

Why you pass a tuple to this function? Wouldn't it be safer to pass a string to avoid IndexErrors?

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8476000 [details] [diff] [review]
bug1040319_tools_v1.patch

Review of attachment 8476000 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/build/versions.py
@@ +26,5 @@
> +    # If version is a function, it will do the bump.
> +    # It takes the old contents as its input to generate
> +    # the new content.
> +    if callable(version): 
> +        return version((contents,))

See my previous comment about passing a string instead of a tuple.

Also, calling "version" doesn't sound right to me... Maybe rename the variable or instead of using bumpFile() call the callable from scripts/release/tag-release.py directly?

Comment 15

[Rail Aliiev [:rail]]

(In reply to Pete Moore [:pete][:pmoore] from comment #12)
> I guess I will also need a similar patch for betas? Would that be
> release-fennec-mozilla-beta.py.template?
> 
> I guess both betas and releases will get made off the esr31 branch(?)
> Hmmmm... What do you think, Rail? Not quite sure how that works.

From what I saw you don't need any changes in other config files.

Comment 16

[Pete Moore [:pmoore][:pete]]

(In reply to Rail Aliiev [:rail] from comment #13)
> Comment on attachment 8475998 [details] [diff] [review]
> bug1040319_buildbot-configs_v1.patch
> 
> Review of attachment 8475998 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Since this "hack" will be going away when stop support for armv6, I'm OK to
> have a function in a config file to avoid pollution of libraries and scripts.

Thanks Rail. This was my thinking too. Then as much of the changes as possible are localised to the esr31 branch, which will die, when it does. This was my thinking too.

> ::: mozilla/release-firefox-mozilla-esr31.py.template
> @@ +42,5 @@
> >  releaseConfig['nextMilestone']       = releaseConfig['nextAppVersion']
> > +
> > +def bumpIntegerInFile(previousContents):
> > +    print previousContents[0]
> > +    return str(int(previousContents[0]) + 1)
> 
> Why you pass a tuple to this function? Wouldn't it be safer to pass a string
> to avoid IndexErrors?

Whoops. That indeed is nonsense - left over mess from debugging/experimenting. Will fix! Sorry, I rushed it out a bit last night as I had to rush off for an appointment, and wanted to just get a sanity check that you were happy with the approach - but yes, very silly wrapping in a tuple! :D

Comment 17

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_buildbot-configs_v2.patch

Hey Rail,

Since we are releasing fennec from esr 31 branch, which we've not done before, I realised we need to create new configs for this.

When setting up my staging environment with a local ship it and release runner instance, I discovered that the first time the staging release runs, release runner seems to require the existence of the previously generated release config file (i.e. the file that is generated from the .template file). For this reason, I have also included staging_release-fennec-mozilla-esr31.py in this patch, even though it is generated from staging_release-fennec-mozilla-esr31.py.template - since I believe release runner requires the file to exist, before it gets generated. If this is not the case, and I'm doing something wrong, let me know!

I've applied your previous suggestions in these patches too.

Again, this is just a feedback request, rather than a full review, since I wanted to show how I was progressing. For example, my buildbot configs reference mozharness single and multi locale configs that I have not generated yet.

I also enabled nightly builds for armv6 as I am guessing this will be needed - although I believe I'll need to make further changes to buildbot configs to get them enabled just on esr31 branch - however, if you think we shouldn't need nightly builds for esr31, and dep builds only are required, let me know!

This is pretty much my first work in buildbot configs, so I wanted some feedback early - I appreciate there may still be some further changes required.

Thanks for your patience and attention! :)

Pete

Comment 18

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_tools_v2.patch

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 8478646 [details] [diff] [review]
bug1040319_buildbot-configs_v2.patch

Review of attachment 8478646 [details] [diff] [review]:
-----------------------------------------------------------------

I overall it looks good, just needs some corrections.

::: mozilla/release-fennec-mozilla-esr31.py.template
@@ +75,5 @@
> +    'build/mozharness': 'production',
> +}
> +
> +# Platform configuration
> +releaseConfig['enUSPlatforms']        = ('android', 'android-x86')

('android-armv6',) I believe

@@ +82,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = ('android',)

android-armv6 here, I think?

@@ +124,5 @@
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {
> +    'android': [

android-armv6?

::: mozilla/staging_release-fennec-mozilla-esr31.py
@@ +1,1 @@
> +# ATTENTION:

Just "touch staging_release-fennec-mozilla-esr31.py" to make release-runner happy. In this case the automatic patch generated by release-runner will look readable.

::: mozilla/staging_release-fennec-mozilla-esr31.py.template
@@ +84,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = ('android',)

android-armv6?

@@ +126,5 @@
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {
> +    'android': [

android-armv6?

Comment 20

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_mozharness_v1.patch

Comment 21

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8478646 [details] [diff] [review]
bug1040319_buildbot-configs_v2.patch

Review of attachment 8478646 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-esr31.py.template
@@ +75,5 @@
> +    'build/mozharness': 'production',
> +}
> +
> +# Platform configuration
> +releaseConfig['enUSPlatforms']        = ('android', 'android-x86')

Quite right - fixed.

@@ +82,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = ('android',)

Indeed - fixed.

@@ +124,5 @@
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {
> +    'android': [

I don't think so here - I checked all other release configs that had the single_locale_options, and all of them specified 'android' and none of them specified e.g. 'android-x86' also when android x86 builds were available for that branch - so I think this may not depend on the chipset architecture(?)

That said, I am only basing this on what I see in other configs, I'm not sure how this config is used.

::: mozilla/staging_release-fennec-mozilla-esr31.py
@@ +1,1 @@
> +# ATTENTION:

This is what I originally tried to do, but I still got errors. However, I'll try again and report back what the error was.

::: mozilla/staging_release-fennec-mozilla-esr31.py.template
@@ +84,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = ('android',)

Thanks - fixed.

@@ +126,5 @@
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {
> +    'android': [

Same as above - I don't think so here.

Comment 22

[Pete Moore [:pmoore][:pete]]

Hey Rail,

I realise after publishing that the comment above looks weird (my responses are to the code, not to your comments) - if you go to https://bugzilla.mozilla.org/page.cgi?id=splinter.html&bug=1040319&attachment=8478646 and click "All Files" you will see that my comments in response to your comments inline.

Thanks!
Pete

Comment 23

[Pete Moore [:pmoore][:pete]]

> > @@ +124,5 @@
> > > +releaseConfig['enableUpdatePackaging']    = False
> > > +releaseConfig['balrog_api_root']          = None
> > > +
> > > +releaseConfig['single_locale_options'] = {
> > > +    'android': [
> > 
> > android-armv6?
> > 
> I don't think so here - I checked all other release configs that had the
> single_locale_options, and all of them specified 'android' and none of them
> specified e.g. 'android-x86' also when android x86 builds were available for
> that branch - so I think this may not depend on the chipset architecture(?)
> 
> That said, I am only basing this on what I see in other configs, I'm not
> sure how this config is used.

OK, I checked the code - and you are quite right. I see here we reference releaseConfig['single_locale_options'][platform] if 'android' is in the platform name:
https://hg.mozilla.org/build/buildbotcustom/file/1dec08be4399/process/release.py#l810
and platform comes from the list in releaseConfig['enUSPlatforms']:
https://hg.mozilla.org/build/buildbotcustom/file/1dec08be4399/process/release.py#l585
which indeed contains 'android-armv6' in our config. So I'll change this too! I should no better than to question! :D Sorry about that.

Pete

Comment 24

[Pete Moore [:pmoore][:pete]]

s/no better/know better/ *eyeroll*

Comment 25

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_buildbot-configs_v3.patch

Feedback incorporated!

Comment 26

[[e-mail] (:bhearsum)]

Comment on attachment 8479346 [details] [diff] [review]
bug1040319_mozharness_v1.patch

Review of attachment 8479346 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/single_locale/release_mozilla-esr_android-armv6.py
@@ +16,5 @@
> +    "buildbot_json_path": "buildprops.json",
> +    "purge_minsize": 10,
> +    "force_clobber": True,
> +    "clobberer_url": "http://clobberer.pvt.build.mozilla.org/index.php",
> +    "locales_file": "buildbot-configs/mozilla/l10n-changesets_mozilla-esr31",

We use JSON files like https://github.com/mozilla/build-buildbot-configs/blob/master/mozilla/l10n-changesets_mobile-release.json for these releases, I think this name needs to be updated. r=me with that fixed.

Comment 27

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_mozharness_v2.patch

Nice spot! Thanks Ben! :)

Comment 28

[Rail Aliiev [:rail]]

Comment on attachment 8479798 [details] [diff] [review]
bug1040319_buildbot-configs_v3.patch

Review of attachment 8479798 [details] [diff] [review]:
-----------------------------------------------------------------

Almost there. See my comments below.

::: mozilla/release-fennec-mozilla-esr31.py.template
@@ +124,5 @@
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {
> +    'android-armv6': [

I looked where this config variable is used.

http://hg.mozilla.org/build/buildbotcustom/file/1dec08be4399/process/release.py#l810 uses it to define single locale extra params for the platforms listed in releaseConfig['l10nPlatforms']. In your case this is ("android-armv6",).
We don't support single locales for armv6, this is why our regular configs redefine releaseConfig['l10nPlatforms'] as ("android",) instead of reusing releaseConfig['enUSPlatforms'].

See also http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/31.0-candidates/build1/android/ vs http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/31.0-candidates/build1/android-armv6/

In your case releaseConfig['l10nPlatforms'] should be set to () (and empty tuple). It won't affect the multilocale build which is used as default APK for Google Play.

Additionally you can delete releaseConfig['single_locale_options'] all together since it's irrelevant for armv6.

::: mozilla/staging_release-fennec-mozilla-esr31.py
@@ +84,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = releaseConfig['enUSPlatforms']

The same here. Needs to be set to ()

@@ +125,5 @@
> +releaseConfig['disablePushToMirrors']     = True
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {

Kill this variable.

::: mozilla/staging_release-fennec-mozilla-esr31.py.template
@@ +84,5 @@
> +releaseConfig['talosTestPlatforms']   = ()
> +releaseConfig['enableUnittests']      = False
> +
> +# L10n configuration
> +releaseConfig['l10nPlatforms']       = releaseConfig['enUSPlatforms']

()

@@ +125,5 @@
> +releaseConfig['disablePushToMirrors']     = True
> +releaseConfig['enableUpdatePackaging']    = False
> +releaseConfig['balrog_api_root']          = None
> +
> +releaseConfig['single_locale_options'] = {

9dd

Comment 29

[Kim Moir [:kmoir] ET]

pmoore: you need a release-fennec-mozilla-esr31.py file in addition to release-fennec-mozilla-esr31.py.template, it is missing in your patches and checkconfig fails on my build dev-master

Comment 30

[Kim Moir [:kmoir] ET]

Also, these patches don't checkconfig on a dev-master. You need to add mozilla-esr31 to the mobile_release_branches in master/master_config.json to enable the platform.  Which means there are tools patches as well in production_masters.json for the associated masters that run mobile release builds.

Comment 31

[Kim Moir [:kmoir] ET]

Attachment: bug1040319bbconfigs.patch

Pete: these are the patches I applied to my dev-master to get the builders for ESR32 for Android to appear.

Also, I had to add links to the new file
[[e-mail] master]$ history | grep ln
 1068  ln -s  /builds/buildbot/kmoir/build3/buildbot-configs/mozilla/staging_release-fennec-mozilla-esr31.py staging_release-fennec-mozilla-esr31.py
 1073  ln -s  /builds/buildbot/kmoir/build3/buildbot-configs/mozilla/release-fennec-mozilla-esr31.py release-fennec-mozilla-esr31.py
 1131  history | grep ln
[[e-mail] master]$ pwd
/builds/buildbot/kmoir/build3/master

and  "mobile_release_branches": ["mozilla-release", "mozilla-beta", "mozilla-esr31" ], in master_config.json.

Comment 32

[Kim Moir [:kmoir] ET]

Sorry previous comment should have said esr31, not esr32.  Also, here is the master with the jobs
 http://dev-master1.srv.releng.scl3.mozilla.com:8036/builders

Next step is apply these patches to your master, attach slaves, and try a staging release. I'll attach patches for the tools repo to include this new branch in production_masters.json

Comment 33

[Kim Moir [:kmoir] ET]

Attachment: bug1040319tools.patch

patch to add mozilla-esr31 branch to masters that build mobile releases

Comment 34

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8480147 [details] [diff] [review]
bug1040319tools.patch

Review of attachment 8480147 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm, thanks Kim!

Comment 35

[Pete Moore [:pmoore][:pete]]

(I've also landed it in https://hg.mozilla.org/users/pmoore_mozilla.com/tools-1040319)

Comment 36

[Pete Moore [:pmoore][:pete]]

(In reply to Kim Moir [:kmoir] from comment #31)
> Created attachment 8480135 [details] [diff] [review]
> bug1040319bbconfigs.patch
> 
> Pete: these are the patches I applied to my dev-master to get the builders
> for ESR32 for Android to appear.
> 
> Also, I had to add links to the new file
> [[e-mail] master]$ history | grep ln
>  1068  ln -s 
> /builds/buildbot/kmoir/build3/buildbot-configs/mozilla/staging_release-
> fennec-mozilla-esr31.py staging_release-fennec-mozilla-esr31.py
>  1073  ln -s 
> /builds/buildbot/kmoir/build3/buildbot-configs/mozilla/release-fennec-
> mozilla-esr31.py release-fennec-mozilla-esr31.py
>  1131  history | grep ln
> [[e-mail] master]$ pwd
> /builds/buildbot/kmoir/build3/master

Thanks Kim. These already existed for me, but I think they are probably created by https://hg.mozilla.org/build/buildbot-configs/file/eedf7a8d2651/setup-master.py#l71 during the buildbot installation (but with an existing installation, the links need to be manually created like you did, as I guess setup-master.py only runs during initial installation).

> and  "mobile_release_branches": ["mozilla-release", "mozilla-beta",
> "mozilla-esr31" ], in master_config.json.

Awesome - I've done this too now.

Thanks again Kim!

Comment 37

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8480135 [details] [diff] [review]
bug1040319bbconfigs.patch

Review of attachment 8480135 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-esr31.py
@@ +39,5 @@
> +#  Repository configuration, for tagging
> +releaseConfig['sourceRepositories']  = {
> +    'mobile': {
> +        'name': 'mozilla-esr31',
> +        'clonePath' : 'releases/mozilla-esr31',

I only found clonePath in staging configs. But I think this file is generated anyway, so it probably does no harm, as the file should get overwritten.

Comment 38

[Lawrence Mandel [:lmandel] (use needinfo)]

Release noted in Firefox 32 as "Android 2.2 and ARMv6 processor chipset no longer supported"

Comment 39

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8475999 [details] [diff] [review]
bug1040319_esr31_branch_v1.patch

Review of attachment 8475999 [details] [diff] [review]:
-----------------------------------------------------------------

Hey Catlee,

If you r+ this, please can you also land for me, as I think I don't have rights to land in-tree.

I've tested building an apk, and that the version code is correctly generated.

I have not tested downloading from the play store with multiple different handsets supporting different architectures (armv6/armv7/x86), to confirm that the correct apk is offered in each case, however Kevin Brosnan will be doing this in QA team, I believe.

Any questions, let me know.

When you are happy with the patch, I have permission from Lawrence Mandel to run 31.1b1 build 1 with these changes.

Thanks,
Pete

Comment 40

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8475999 [details] [diff] [review]
bug1040319_esr31_branch_v1.patch

[Approval Request Comment]
If this is not a sec:{high,crit} bug, please state case for ESR consideration:
User impact if declined: No armv6 builds of fennec
Fix Landed on Version: not yet pushed
Risk to taking this patch (and alternatives if risky): none - currently no fennec builds on esr31 branch so worst case, fennec builds fail and nothing is lost
String or UUID changes made by this patch: no l10n changes with this patch

See https://wiki.mozilla.org/Release_Management/ESR_Landing_Process for more info.

Comment 41

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8475999 [details] [diff] [review]
bug1040319_esr31_branch_v1.patch

Approved for ESR31.

Comment 43

[Pete Moore [:pmoore][:pete]]

So I have some good news!

Staging runs ran through to completion:
http://dev-stage01.srv.releng.scl3.mozilla.com/pub/mozilla.org/mobile/candidates/31.1.0esr-candidates/build12/android-armv6/multi/

Please note these are *staging* builds signed with staging keys. I'm now going to be landing all the changes to our production repos, and creating the official 31.1.0esr release with the correct signing keys and correct locales, etc.

Another useful validation is that the manifest indeed seems to be getting updated correctly, e.g.:

pmoore@Elisandra:~ $ ssh [e-mail] head -10 /builds/slave/rel-m-esr31-and-a6_bld-0000000/build/obj-firefox/mobile/android/base/AndroidManifest.xml
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
      package="org.mozilla.firefox"
      android:installLocation="auto"
      android:versionCode="2014080006" <-- the new versionCode
      android:versionName="31.1.0"
      android:sharedUserId="org.mozilla.firefox.sharedID"
      >
    <uses-sdk android:minSdkVersion="8"
              android:targetSdkVersion="17"/>
pmoore@Elisandra:~ $ 

and this perfectly corresponds to it being the 6th iteration of the build, i.e. it was generated by:
http://dev-master1.srv.releng.scl3.mozilla.com:8444/builders/release-mozilla-esr31-android-armv6_build/builds/6

Therefore it appears to be correctly bumping the versionCode by one with each release made.

This can also be seen here:
https://hg.mozilla.org/users/pmoore_mozilla.com/mozilla-esr31/rev/cf91eff58905

So both parts are working correctly:
1) Counter is getting incremented with each release
2) Release process is bumping the file in source control, so the increase will not be "forgotten"

I'll update this bug with the latest patches for all the repos. It may take a couple of hours to prepare this, as I will need to strip out all the staging config changes in the process, where they were necessary.

Please note this is a multilocale build, there are not individual locales. It is my understanding that this is what the requirements are.

Comment 44

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_mozharness_v3.patch

Updated version based on what was successfully used in staging.

Comment 45

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8480147 [details] [diff] [review]
bug1040319tools.patch

Marking as obsolete as it is included in a later aggregated patch in this bug.

Comment 46

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8480135 [details] [diff] [review]
bug1040319bbconfigs.patch

Marking as obsolete since it is included in a later aggregated patch in this bug.

Comment 47

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_buildbot-configs_v4.patch

This is the "normalised" patch I get after removing staging-specific parts in files that are also used in prod - i.e. a cleaned up version of the delta I had in my staging environment, stripped of the hacks needed to get staging working.

Comment 48

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8484342 [details] [diff] [review]
bug1040319_mozharness_v3.patch

Ben/Rail - adding you both so whoever gets there first can take it.

Thanks guys!
Pete

Comment 49

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8484406 [details] [diff] [review]
bug1040319_buildbot-configs_v4.patch

And this one too...

Comment 50

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8475999 [details] [diff] [review]
bug1040319_esr31_branch_v1.patch

I'm not able to land this change - can someone land this for me?

pmoore@Elisandra:/Volumes/TOSHIBA EXT/mozilla-esr31 $ hg out
*** failed to import extension hggit: No module named dulwich.errors
comparing with ssh://hg.mozilla.org/releases/mozilla-esr31/
searching for changes
changeset:   200301:6ae5e6c8ede9
tag:         tip
user:        Peter Moore <[e-mail]>
date:        Thu Sep 04 23:12:46 2014 +0200
summary:     Bug 1040319 - Ensure that Fennec builds from mozilla-esr31 have a buildID to allow for armv6/Android 2.2 users to update to mozilla-esr31 apks,r=catlee,a=lmandel

pmoore@Elisandra:/Volumes/TOSHIBA EXT/mozilla-esr31 $ hg push
*** failed to import extension hggit: No module named dulwich.errors
pushing to ssh://hg.mozilla.org/releases/mozilla-esr31/
searching for changes
remote: abort: could not lock repository /repo/hg/mozilla/releases/mozilla-esr31/: Permission denied
abort: unexpected response: empty string
pmoore@Elisandra:/Volumes/TOSHIBA EXT/mozilla-esr31 $

Comment 51

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_esr31_branch_v2.patch

Same patch as before, but now with commit headers etc (generated with hg export) so that somebody can land it for me (I don't have permission).

Thanks,
Pete

Comment 52

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8478647 [details] [diff] [review]
bug1040319_tools_v2.patch

Review of attachment 8478647 [details] [diff] [review]:
-----------------------------------------------------------------

This was previously a feedback+ so adding a r? for landing it.

Thanks!
Pete

Comment 53

[Pete Moore [:pmoore][:pete]]

Please note: as soon as all 4 patches land and buildbot-configs and mozharness changes are merged to production branches, the official Fennec 31.1.0esr build 1 can be triggered in ship it (reconfig will be done automatically by release runner).

Comment 54

[Rail Aliiev [:rail]]

Comment on attachment 8484342 [details] [diff] [review]
bug1040319_mozharness_v3.patch

Review of attachment 8484342 [details] [diff] [review]:
-----------------------------------------------------------------

lgtm

Comment 55

[Lawrence Mandel [:lmandel] (use needinfo)]

Comment on attachment 8484509 [details] [diff] [review]
bug1040319_esr31_branch_v2.patch

pmoore added Hg headers as he wasn't able to land himself. Adding approval back to the patch.

Comment 56

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8478647 [details] [diff] [review]
bug1040319_tools_v2.patch

https://hg.mozilla.org/build/tools/rev/f1861b490891

Comment 57

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8484342 [details] [diff] [review]
bug1040319_mozharness_v3.patch

default:    https://hg.mozilla.org/build/mozharness/rev/17032ad63c96
production: https://hg.mozilla.org/build/mozharness/rev/63d2fe5bfca7

Comment 58

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr31/rev/8bba0395e567

Comment 59

[Rail Aliiev [:rail]]

Comment on attachment 8484406 [details] [diff] [review]
bug1040319_buildbot-configs_v4.patch

Review of attachment 8484406 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/release-fennec-mozilla-esr31.py
@@ +30,5 @@
> +#  Current version info
> +releaseConfig['version']             = '31.1.0esr'
> +releaseConfig['appVersion']          = '31.1.0'
> +releaseConfig['milestone']           = releaseConfig['appVersion']
> +releaseConfig['buildNumber']         = 12

A nit. Can you make it 1 instead of 12, just to have better incremental diffs.

Comment 60

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8484406 [details] [diff] [review]
bug1040319_buildbot-configs_v4.patch

default:     https://hg.mozilla.org/build/buildbot-configs/rev/302b01433b36
production:  https://hg.mozilla.org/build/buildbot-configs/rev/d27209ad26aa

Comment 61

[Pete Moore [:pmoore][:pete]]

Thanks Rail for all your reviews! =)
Thanks Ryan for landing!

Will wait for jenkins to run tests for buildbot-configs changes, and then I will trigger Fennec 31.1.0esr build 1 on production ship it.

Please note, mobile tests have not yet been enabled - I'll create a bug for this so we can set it up asap.

Pete

Comment 62

[Pete Moore [:pmoore][:pete]]

(In reply to Pete Moore [:pete][:pmoore] from comment #45)
> Comment on attachment 8480147 [details] [diff] [review]
> bug1040319tools.patch
> 
> Marking as obsolete as it is included in a later aggregated patch in this
> bug.

Whoops - no it wasn't. Setting back.

Comment 63

[Pete Moore [:pmoore][:pete]]

Comment on attachment 8480147 [details] [diff] [review]
bug1040319tools.patch

Landed: https://hg.mozilla.org/build/tools/rev/b6042cbc7699

Comment 64

[Pete Moore [:pmoore][:pete]]

From :kbrosnan:

I ran our smoke tests against the builds, there were some build problems and test failures https://moztrap.mozilla.org/runtests/run/5327/env/26/

First of this is a release Firefox for Android build (org.mozilla.firefox). We cannot ship it to Firefox Beta (org.mozilla.firefox_beta) users as we had planned.

* No release notes, www.mozilla.org/en-US/mobile/31.1.0/releasenotes
* Droid Incredible (2.2) YouTube app seems to be out of date, thus clicking on a m.youtube.com link presents an error message from the Google Youtube app. I do not believe this to be a Firefox issue. No updates for YT available in the Play Store. Using an Samsung SCH-R720 running Android 2.3.4 is able to launch the YouTube app correctly.
* Allowing popups fails to work need to check this behavior vs 31 release

Kevin Brosnan

Comment 65

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_esr31_branch_beta_branding_v1.patch

Hi Rail,

Thanks for your tip about reversing http://hg.mozilla.org/releases/mozilla-release/rev/a3cc3e46b571 to force the build system to use: http://hg.mozilla.org/releases/mozilla-release/file/a3cc3e46b571/mobile/android/branding/beta/configure.sh - this is what I have done in this patch.

Many thanks,
Pete

Comment 66

[Rail Aliiev [:rail]]

The patch itself looks good to me, I have a couple of concerns regarding the procedure:

* If the change is a one off, we could probably build manually, without landing the patch. This is a bit error-prone. :/

* The automation will be generating another build under 31.x.x-candidates directory, while the binaries are more like 31.x.xbx-candidates. Maybe we should move the files somewhere else manually if we decide to use automation to build this.

* If we decide to use this patch and our release automation to generate the builds, I'd land the patch on a separate branch (not default), so we don't even care about backing it out. Branching off of MOBILE3110esr_2014090505_RELBRANCH sounds like a candidate.

What is the plan here?

Comment 67

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #66)
> The patch itself looks good to me, I have a couple of concerns regarding the
> procedure:
> 
> * If the change is a one off, we could probably build manually, without
> landing the patch. This is a bit error-prone. :/
> 
> * The automation will be generating another build under 31.x.x-candidates
> directory, while the binaries are more like 31.x.xbx-candidates. Maybe we
> should move the files somewhere else manually if we decide to use automation
> to build this.

Primarily for this reason, I'm pretty in favour of building by hand. The builds are not terribly difficult, and it's easier not to clutter up FTP and the repository with stuff for a one-off IMO. There's no reason for these builds to ever be on FTP as far as I can tell - we just need them for a test in the Google Play store.

> 
> * If we decide to use this patch and our release automation to generate the
> builds, I'd land the patch on a separate branch (not default), so we don't
> even care about backing it out. Branching off of
> MOBILE3110esr_2014090505_RELBRANCH sounds like a candidate.

+1, if we go with automation.

Comment 68

[Kevin Brosnan [Ex-Mozilla]]

The beta build is a one off.

Comment 69

[Rail Aliiev [:rail]]

Comment on attachment 8490842 [details] [diff] [review]
bug1040319_esr31_branch_beta_branding_v1.patch

We agreed on IRC to proceed with a manual one off build to avoid cluttering repo/ftp.

Comment 70

[Pete Moore [:pmoore][:pete]]

Running a manual build on dev-linux64-ec2-pmoore.dev.releng.use1.mozilla.com

Comment 71

[Pete Moore [:pmoore][:pete]]

The build has worked, quite miraculously! I'm just at the closing stages now, I'll attach the script I used when I've got to the end...

I generated the script with the help of my good friend sed, and it pretty much worked off the bat (although took an age to actually build): https://hg.mozilla.org/build/braindump/raw-file/fc6ff19493c1/buildbot-related/scrape_buildbot_logs_for_commands.sed

Will update shortly with details about location of new apk.

Comment 72

[Pete Moore [:pmoore][:pete]]

Uploaded to:

/home/pmoore/fennec-31.1.0esr.en-US.android-arm-armv6.apk

on

people.mozilla.org

Comment 73

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_build_script.sh

This script was generated with:

curl -L 'http://ftp.mozilla.org/pub/mozilla.org/mobile/candidates/31.1.0esr-candidates/build1/logs/release-mozilla-esr31-android-armv6_build-bm85-build1-build0.txt.gz' 2>/dev/null | gunzip | ./scrape_buildbot_logs_for_commands.sed > all_commands.sh

where scrape_buildbot_logs_for_commands.sed is:
https://hg.mozilla.org/build/braindump/raw-file/fc6ff19493c1/buildbot-related/scrape_buildbot_logs_for_commands.sed

And then any tweaks I needed to make (steps to skip, for example, or creating the initial directories) I have added with comments, so it is straightforward to see the deviations I added to the original run script.

Essentially it skips the upload steps, and after the package has been made (make package), all subsequent steps are skipped.

Additionally, after esr31 repo has been checked out, the script patches the repo with the fix from comment 65.

Lastly, bhearsum showed me how to manually generate a signing token (many thanks!) so that we could get the signing to work.

I ran chmod 644 on the apk on people.mozilla.org in my home directory so that it can be downloaded by anyone with access to people, to avoid confusing matters with firefox beta apks being uploaded to ftp.

Comment 74

[Rail Aliiev [:rail]]

Comment on attachment 8492346 [details]
bug1040319_build_script.sh

>python /builds/slave/rel-m-esr31-and-a6_bld-0000000/tools/buildfarm/utils/retry.py -s 1 -r 5 -t 1260 bash -c 'if [ -f "mobile/android/config/mozconfigs/android-armv6/release" ]; then                        echo Using in-tree mozconfig;                        cp mobile/android/config/mozconfigs/android-armv6/release .mozconfig;                    else                        echo Downloading mozconfig;                        wget -O .mozconfig https://hg.mozilla.org/build/buildbot-configs/raw-file/FENNEC_31_1_0esr_RELEASE/mozilla2/android-armv6/mozilla-esr31/release/mozconfig;                    fi'
>cd /builds/slave/rel-m-esr31-and-a6_bld-0000000/build 
>cat .mozconfig
...

># here we patch the build with the tweaks we need to make it work with firefox beta
>curl -L 'https://bug1040319.bugzilla.mozilla.org/attachment.cgi?id=8490842' | patch -p1 -i -

You should move the patching part just after "hg up". It should be applied before you copy the used mozconfig to .mozconfig. Otherwise you'll have the same product ID.

Comment 75

[Pete Moore [:pmoore][:pete]]

Attachment: bug1040319_build_script.sh

Thanks for taking care of everything yesterday Rail! Updated version attached, in case we need it again...

Thanks,
Pete

Comment 76

[Rail Aliiev [:rail]]

Comment on attachment 8494299 [details]
bug1040319_build_script.sh

Yup, exactly like that.

Comment 77

[Lawrence Mandel [:lmandel] (use needinfo)]

Pete - I think we're done here and this bug can be resolved. Can you please confirm.

Comment 78

[Pete Moore [:pmoore][:pete]]

Quite right! Thanks Lawrence.