[PATCH 2/3 Try#2] NOR Flash Support (U-Boot env)

[Harald Welte]

[moving this thread to the u-boot list, since it's not a kernel issue]

Adapt default env to boot from /dev/mtdblock5 and specify
physmap-flash for nor at start of kernel commandline

I think those changes are fine, however: devirginator has to support
both GTA01 and GTA02, therefore we can't unconditionally make such
changes.

I think first devirginator will have to be modified to use different
environment.in partitions depending on the actual model
(GTA01/GTA02/hxd8).

Werner, what are your thoughts on this? How did you solve it for hxd8?
Plase don't tell me "fork the codebase"?

[Andy Green]

Post by Harald Welte
[moving this thread to the u-boot list, since it's not a kernel issue]

Adapt default env to boot from /dev/mtdblock5 and specify
physmap-flash for nor at start of kernel commandline

I think those changes are fine, however: devirginator has to support
both GTA01 and GTA02, therefore we can't unconditionally make such
changes.
I think first devirginator will have to be modified to use different
environment.in partitions depending on the actual model
(GTA01/GTA02/hxd8).
Werner, what are your thoughts on this? How did you solve it for hxd8?
Plase don't tell me "fork the codebase"?

Hi Harald -

If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up. The MTD ROM support should
accept it despite the Flash probe would fail, so it is one way to get a
unified behaviour.

On the more general question of how to manage GTA-01 when it is not
directly a consideration for reaching production of GTA-02, there's
another model for coming at this.

We could treat the GTA-01 low level software like kernel, U-Boot and
devirginator (I guess generally usermode code doesn't care much what it
runs on) as "released" for GTA-01 and instead of trying to target -01
and -02 for new development work -- which focuses on a GTA-02 that can
go to production -- we find or fund a guy interested to manage
backporting newer GTA-02 work on to a "stable" GTA-01 tree -- it's a bit
like the 2.6 stable branch kernel maintainer.

If we additionally consider a GTA-03 that is the subject of occasional
fantasies I read about, there is a clear problem of scaling the
development effort over multiple variants (and in my newbie point of
view, having to deal with intricacies of history) when really only the
"next" version can pay off the work in terms of going to production, and
every stumble or blocking that comes from GTA-01 retards GTA-02
production in the narrowest sense needlessly. All the work is GPL so
there is no issue exposing in realtime everything we plan to give out
binaries for.

What are your thoughts?

- -Andy

[Werner Almesberger]

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up.

Hmm, and people installing the new kernel on GTA01 like they
always did so far would find themselves with an unbootable system.
Not nice :-(

Post by Andy Green
there is a clear problem of scaling the
development effort over multiple variants

Our ultimate goal is to merge all our code into the respective
upstream sources, so creating permanent branches would actually
move us away from where we want to be, and make life harder for
everyone concerned.

Since the GTA01, GTA02, and whatever code has to co-exist in the
same tree in the end anyway, it's better to do this right from
the beginning.

So the question would be if we can disable the NOR partition at
run time (i.e., when running on GTA01), or, failing that, we can
leave it as a compile-time option.

It's nice if one doesn't have to worry about matching builds with
the exact hardware version. With u-boot, we've lost that a long
time ago (actually, even before I got involved), and have suffered
for it often enough. So let's keep things sane and simple at least
for the kernel.

For the devirginator, three ideas come to mind:

1) Just use a different set of configuration files. They're quite
simple, so branching these wouldn't be all that much of a pain.

2) Implement #ifdef-like functionality in envedit.pl and pass these
items on the command line. That way, environment.in can stay the
same.

3) Like 2), and add an environment variable with the hardware version
to u-boot. That way, envedit.pl could implement the #ifdef-like
functionality with only looking at the environment obtained from
the machine.

I'll make the changes for 2) and 3) to envedit.pl. They look like
something that's good to have anyway. Then we can think about what
we want to do on the u-boot side.

- Werner

[Andy Green]

Post by Werner Almesberger

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up.

Hmm, and people installing the new kernel on GTA01 like they
always did so far would find themselves with an unbootable system.
Not nice :-(

They can DFU-update to a contemporary GTA-01 U-Boot and they're away
again, if I understood you.

Post by Werner Almesberger

Post by Andy Green
there is a clear problem of scaling the
development effort over multiple variants

Our ultimate goal is to merge all our code into the respective
upstream sources, so creating permanent branches would actually
move us away from where we want to be, and make life harder for
everyone concerned.

My point is that the even more ultimate goal is to get a production
version out. For example I don't have a GTA-01 and I can't test what I
am putting out for GTA-01... should that really delay what can usefully
be done towards GTA-02? These kind of thoughts lead me to this
backporting concept.

Post by Werner Almesberger
2) Implement #ifdef-like functionality in envedit.pl and pass these
items on the command line. That way, environment.in can stay the
same.
3) Like 2), and add an environment variable with the hardware version
to u-boot. That way, envedit.pl could implement the #ifdef-like
functionality with only looking at the environment obtained from
the machine.
I'll make the changes for 2) and 3) to envedit.pl. They look like
something that's good to have anyway. Then we can think about what
we want to do on the u-boot side.

If the future is kind that can work fine. Or it can turn into an
impenetrable #ifdef forest :-)

- -Andy

[Werner Almesberger]

Post by Andy Green
They can DFU-update to a contemporary GTA-01 U-Boot and they're away
again, if I understood you.

Not quite. They mainly have to edit and update their environment.
If Linux user space references any partitions, that would have to
be edited as well.

So this would be the type of change that causes all sorts of upset.

Post by Andy Green
My point is that the even more ultimate goal is to get a production
version out.

Sure, but if you create weeks of extra work to save a couple of
hours, that's not good either.

Post by Andy Green
For example I don't have a GTA-01 and I can't test what I
am putting out for GTA-01...

Well, just don't actively break anything. It's easy to fix things
that might slip through. It's hard to fix things if the breakage
has become a design goal.

Post by Andy Green
If the future is kind that can work fine. Or it can turn into an
impenetrable #ifdef forest :-)

I'm looking forward to the day when the devirginator has to support
a dozen different product lines ;-))

The enhanced envedit.pl is in SVN revision 3729. I've also added
macro expansion, just in case.

- Werner

[Andy Green]

Post by Werner Almesberger

Post by Andy Green
They can DFU-update to a contemporary GTA-01 U-Boot and they're away
again, if I understood you.

Not quite. They mainly have to edit and update their environment.

They are in a position to do this with minicom. In short they need to
read a wiki page about it, but it doesn't brick them.

Post by Werner Almesberger
If Linux user space references any partitions, that would have to
be edited as well.

On the userspace that I inherited from Willie, which I think is pretty
common, only /dev/root -> / is used, ie, no changes to /etc/fstab are
required once the U-Boot -> kernel commandline magic is done, userspace
just comes up fine. BTW the U-Boot envs I see on the boards I got bear
zero relationship to your environment.in, so environment.in is not
getting tested in my (limited) experience.

Post by Werner Almesberger

Post by Andy Green
My point is that the even more ultimate goal is to get a production
version out.

Sure, but if you create weeks of extra work to save a couple of
hours, that's not good either.

Inbetween our causing some poor guy "weeks" of work while we eat grapes
peeled for us by beautiful maidens; and our trying to shoulder the
weight of multiple worlds while staggering forwards: there is a sweet
spot where we can stride briskly on and the load on the GTA-01
backporter is light.

Post by Werner Almesberger

Post by Andy Green
For example I don't have a GTA-01 and I can't test what I
am putting out for GTA-01...

Well, just don't actively break anything. It's easy to fix things
that might slip through. It's hard to fix things if the breakage
has become a design goal.

Breaking GTA-01 per se doesn't buy me anything, my point is rather that
ignoring it might, when an able GTA-01 person has our continuous working
GTA-02 patches to start from.

Post by Werner Almesberger

Post by Andy Green
If the future is kind that can work fine. Or it can turn into an
impenetrable #ifdef forest :-)

I'm looking forward to the day when the devirginator has to support
a dozen different product lines ;-))

I would get a case of brandy just in case, either way it is going to
help ;-)

- -Andy

[Harald Welte]

Hi Andy

Post by Andy Green

Post by Werner Almesberger

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up.

Hmm, and people installing the new kernel on GTA01 like they
always did so far would find themselves with an unbootable system.
Not nice :-(

They can DFU-update to a contemporary GTA-01 U-Boot and they're away
again, if I understood you.

updating the bootloader is not something that people easily are willing
to do, especially if they don't have a debug board to recover. Go and
ask about the experience of existing users :/

Also, since old u-boot also doesn't support new kernel, I consider this
an absolutely unacceptable situation. If there are such dependencies,
the system needs to fail gracefully, i.e. at least inform the user
on-screen aobut what he is supposed to respove this.

Given the state of the software, many people switch back and forth
between kernel/rootfs revisions in order to test for regressions, etc.
You cannot put additional obstacles in there.

Post by Andy Green

Post by Werner Almesberger
Our ultimate goal is to merge all our code into the respective
upstream sources, so creating permanent branches would actually
move us away from where we want to be, and make life harder for
everyone concerned.

My point is that the even more ultimate goal is to get a production
version out. For example I don't have a GTA-01 and I can't test what I
am putting out for GTA-01... should that really delay what can usefully
be done towards GTA-02? These kind of thoughts lead me to this
backporting concept.

Please see my other posting. If I was a GTA01 customer, I would be
seriously upset about OM dropping hardware support of it before it was
ever finished!!!

And if you don't have a GTA01, then you should make OM provide you one
ASAP...

[Harald Welte]

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up. The MTD ROM support should
accept it despite the Flash probe would fail, so it is one way to get a
unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01 partition
tables have to be as-is.

Post by Andy Green
We could treat the GTA-01 low level software like kernel, U-Boot and
devirginator (I guess generally usermode code doesn't care much what it
runs on) as "released" for GTA-01 and instead of trying to target -01
and -02 for new development work -- which focuses on a GTA-02 that can
go to production -- we find or fund a guy interested to manage
backporting newer GTA-02 work on to a "stable" GTA-01 tree -- it's a bit
like the 2.6 stable branch kernel maintainer.

I don't think this is acceptable. People bought GTA01 in the
expectation that we would once finish the software stack for it. If OM
doesn't deliver a working phone software stack to them, they will feel
tricked into buying a device based on wrong pretext/suggestions. It
sends a really bad message to the community.

So for the time being, I'd think it is not acceptable to drop GTA01
support from anything. All features (unless hardware like
wifi/accelerometer/... is required) have to be en par on both units.
Also, there is no technical reason why they'd not be.

Cheers,

[Michael 'Mickey' Lauer]

Post by Harald Welte

Post by Andy Green
We could treat the GTA-01 low level software like kernel, U-Boot and
devirginator (I guess generally usermode code doesn't care much what it
runs on) as "released" for GTA-01 and instead of trying to target -01
and -02 for new development work -- which focuses on a GTA-02 that can
go to production -- we find or fund a guy interested to manage
backporting newer GTA-02 work on to a "stable" GTA-01 tree -- it's a bit
like the 2.6 stable branch kernel maintainer.

I don't think this is acceptable. People bought GTA01 in the
expectation that we would once finish the software stack for it. If OM
doesn't deliver a working phone software stack to them, they will feel
tricked into buying a device based on wrong pretext/suggestions. It
sends a really bad message to the community.

I agree wholeheartedly and I have also said this at other occasions.
We can't afford to "pull a Nokia" [1] here!

[1] Nokia 770 Internet Tablet which has been abandoned when the N800
came out.

Regards,

:M:

[Andy Green]

Post by Harald Welte

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up. The MTD ROM support should
accept it despite the Flash probe would fail, so it is one way to get a
unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01 partition
tables have to be as-is.

Fine. This is actually compatible with the release/backport thing I
proposed.

Post by Harald Welte

Post by Andy Green
We could treat the GTA-01 low level software like kernel, U-Boot and
devirginator (I guess generally usermode code doesn't care much what it
runs on) as "released" for GTA-01 and instead of trying to target -01
and -02 for new development work -- which focuses on a GTA-02 that can
go to production -- we find or fund a guy interested to manage
backporting newer GTA-02 work on to a "stable" GTA-01 tree -- it's a bit
like the 2.6 stable branch kernel maintainer.

I don't think this is acceptable. People bought GTA01 in the

"acceptable" for whom is a cogent question here.

Post by Harald Welte
expectation that we would once finish the software stack for it. If OM
doesn't deliver a working phone software stack to them, they will feel
tricked into buying a device based on wrong pretext/suggestions. It
sends a really bad message to the community.

The task that seems most important to me is to get GTA-02 in shape for
production so our customer is able to go on. If we fought the Good
Fight for GTA-01, updated it daily but the customer is not able to make
a product, it would be a failure on our part I think. In the scenario I
outline, we are constantly handing the GTA-01 guy(s) the pieces they
need to continue a living codebase. In fact there is a GTA-02 BECAUSE
the customer does not want to go to bat with GTA-01.

Post by Harald Welte
So for the time being, I'd think it is not acceptable to drop GTA01
support from anything. All features (unless hardware like
wifi/accelerometer/... is required) have to be en par on both units.
Also, there is no technical reason why they'd not be.

Absolutely, no reason the patches can't be used by GTA-01. But it seems
the customer will not be making GTA-01s.

- -Andy

[Harald Welte]

Post by Andy Green

Post by Harald Welte

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also has
the NOR patch, it will create a (useless, but logically present) MTD
device representing the (IIRC, on GTA-01 nonexistent) NOR device's
footprint and make the partitions match up. The MTD ROM support should
accept it despite the Flash probe would fail, so it is one way to get a
unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01 partition
tables have to be as-is.

Fine. This is actually compatible with the release/backport thing I
proposed.

Go and ask the community. I do not think they will be happy if they are
treated as second-class citizens with some volunteers backporting stuff
that OM (the company) does.

OM has to live up to its own promises.

Also, be assured that maintaining two kernel/uboot (and even rootfs?)
codebases i.e. forking the code will create a maintenance nightmare and
way more work further down the road.

That's exactly why people want to merge their code into the 'mainline'
tree. Avoid fragmentation. Keeping multiple trees in sync is an
utterly boring and dull task. A neverending one, too.

Post by Andy Green

Post by Harald Welte

Post by Andy Green
We could treat the GTA-01 low level software like kernel, U-Boot and
devirginator (I guess generally usermode code doesn't care much what it
runs on) as "released" for GTA-01 and instead of trying to target -01
and -02 for new development work -- which focuses on a GTA-02 that can
go to production -- we find or fund a guy interested to manage
backporting newer GTA-02 work on to a "stable" GTA-01 tree -- it's a bit
like the 2.6 stable branch kernel maintainer.

I don't think this is acceptable. People bought GTA01 in the

"acceptable" for whom is a cogent question here.

acceptable to my own expectations how a company/project that 'knows
better what to not do' will treat its most valuable first customers.

Post by Andy Green
Absolutely, no reason the patches can't be used by GTA-01. But it seems
the customer will not be making GTA-01s.

poeple are devirginating their devices as a last-resort if they are
bricked. This is even more important on GTA01 (no nor emergency boot)
than on GTA02.

[Mike Montour]

Post by Harald Welte
poeple are devirginating their devices as a last-resort if they are
bricked. This is even more important on GTA01 (no nor emergency boot)
than on GTA02.

There is now a manual unbricking procedure
(http://wiki.openmoko.org/wiki/Neo1973_Debug_Board_v2/Unbricking). As a
member of the GTA01-owners community, I think that it would be
acceptable to drop GTA01 support from the devirginator (but not from
u-boot or the kernel) if it would speed up the GTA02 development process.

[Werner Almesberger]

Post by Mike Montour
There is now a manual unbricking procedure
(http://wiki.openmoko.org/wiki/Neo1973_Debug_Board_v2/Unbricking).

One problem with GTA01 is that not everyone has a debug board, so
quite a few people have good reasons to be skittish about anything
that might brick their device.

GTA02 will be considerably harder to brick. The debug board (v3)
will basically act as a key to even allow you to brick it. (That
is, erase the NOR.)

Post by Mike Montour
I think that it would be acceptable to drop GTA01 support from the
devirginator

The devirginator isn't such a big concern for me either. However,
forcing people to update u-boot or to make non-trivial changes to
their environment and/or user space would be. And in this case,
the upgrade would even have to be undone if you go back to an
older kernel, which would be a little nightmare in its own right.

In any case, I don't think there is any valid technical reason
why we would need to cause this sort of disruptions.

- Werner

[Harald Welte]

Post by Werner Almesberger
In any case, I don't think there is any valid technical reason
why we would need to cause this sort of disruptions.

Exactly.

[Graeme Gregory]

On Mon, 24 Dec 2007 15:32:42 +0100

Post by Harald Welte

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also
has the NOR patch, it will create a (useless, but logically
present) MTD device representing the (IIRC, on GTA-01 nonexistent)
NOR device's footprint and make the partitions match up. The MTD
ROM support should accept it despite the Flash probe would fail, so
it is one way to get a unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01 partition
tables have to be as-is.

I can see this playing havoc with the upgrade kernel by ipkg stuff.

Graeme

[Andy Green]

Post by Graeme Gregory
On Mon, 24 Dec 2007 15:32:42 +0100

Post by Harald Welte

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also
has the NOR patch, it will create a (useless, but logically
present) MTD device representing the (IIRC, on GTA-01 nonexistent)
NOR device's footprint and make the partitions match up. The MTD
ROM support should accept it despite the Flash probe would fail, so
it is one way to get a unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01 partition
tables have to be as-is.

I can see this playing havoc with the upgrade kernel by ipkg stuff.

Is that really beyond us?

$ cat /proc/mtd | grep kernel | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock3
$ cat /proc/mtd | grep rootfs | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock5

- -Andy

[Graeme Gregory]

On Wed, 26 Dec 2007 17:12:46 +0000

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Graeme Gregory
On Mon, 24 Dec 2007 15:32:42 +0100

Post by Harald Welte

Post by Andy Green
If the kernel that devirginator drops in the device on GTA-01 also
has the NOR patch, it will create a (useless, but logically
present) MTD device representing the (IIRC, on GTA-01 nonexistent)
NOR device's footprint and make the partitions match up. The MTD
ROM support should accept it despite the Flash probe would fail,
so it is one way to get a unified behaviour.

this is not good. Since GTA01 is already deployed, I think it will
generate a nightmare for updating and compatibility. GTA01
partition tables have to be as-is.

I can see this playing havoc with the upgrade kernel by ipkg stuff.

Is that really beyond us?
$ cat /proc/mtd | grep kernel | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock3
$ cat /proc/mtd | grep rootfs | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock5

No, but what if its the wrong kernel version, ie you flash the one
without NOR in a nor u-boot, or the one with NOR in the non nor
u-boot.

Graeme

[Andy Green]

Post by Graeme Gregory

Post by Andy Green

Post by Graeme Gregory
I can see this playing havoc with the upgrade kernel by ipkg stuff.

Is that really beyond us?
$ cat /proc/mtd | grep kernel | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock3
$ cat /proc/mtd | grep rootfs | cut -d':' -f1 | sed \
s/mtd/\\/dev\\/mtdblock/g
/dev/mtdblock5

No, but what if its the wrong kernel version, ie you flash the one
without NOR in a nor u-boot, or the one with NOR in the non nor
u-boot.

Well you can tell about the U-Boot you booted from:

$ cat /proc/cmdline | grep physmap

will give ! -z result for the "nor U-Boot"

What happens at the time you upgrade the U-Boot and Kernel packages can
be handled by a "requires" in the "nor kernel" for the "nor U-Boot"
package, and vice versa, so they have to go in together.

If the nor-less GTA-01 just never sees any of this stuff because it is
on its own branch (as, if I understood Werner, is already the case for
U-Boot) and GTA-02 goes out with a kernel-gta02 package for example that
won't be compatible with whatever the GTA-01 kernel package is called,
that sounds good to me too.

If someone can figure out a way to get the NOR probed last (meddle with
the kernel Makefile order?) that will also solve it -- and need another
round of patches.

- -Andy

[Werner Almesberger]

Post by Andy Green
If the nor-less GTA-01 just never sees any of this stuff because it is
on its own branch (as, if I understood Werner, is already the case for
U-Boot)

It's not a separate branch of the code base. It's just a different
configuration of u-boot.

(The one you set with "make ARCH=arm gta02v4_config" or similar.)

- Werner

[Werner Almesberger]

Post by Harald Welte
How did you solve it for hxd8?
Plase don't tell me "fork the codebase"?

Bingo ! :-) Well, in HXD8, the devirginator is mainly related to
the production environment, which is product-specific anyway, so
finding that fork didn't bother me too much.

- Werner