Directory Service for publicly-accessible #FujiNet sites

[SenorRossie]

1 hour ago, x=usr(1536) said:

5 hours ago, SenorRossie said:

Wouldn't a 'simple' ddns (RFC2136 style) client simplify things? If needed you could split it into official and user contributed content.

 

Just my $.02

Maybe.  I gave the idea some thought earlier on, and am not opposed to it in concept, but my position is that there needs to be no distinction between hosts as being official or otherwise.  The one possible exception of having DNS-style root servers that help to replicate directory data globally might be necessary, but, much like the Internet at large, I feel it's important to keep this as unmanaged as possible.

The distinction between 'official' and user contributed hosts is entirely optional and could be as simple as using a different (sub-) domain.

A pro for dns is that it has been a part of the internet for a long time, has been well documented and is cross platform. The DDNS part is also widely in use and well supported.

 

DNS is used for registration and storing of host records, added bonus: Global distributed storage. A second step could be axfr'ing (or ixfr if using slaves*) the zone to one or more central systems, checking the records and publishing them in a format easily accessible/parsable by the fujinet device.

 

* When using slaves to generate verified lists, an automated solution is feasable where an ixfr triggers regeneration of the published list.

 

[+x=usr(1536)]

38 minutes ago, SenorRossie said:

The distinction between 'official' and user contributed hosts is entirely optional and could be as simple as using a different (sub-) domain.

What is the benefit of having this kind of separation?  Maybe I'm missing something in your intent behind the idea, but I'm not clear as to how having that separation would be necessary or desirable.

38 minutes ago, SenorRossie said:

A pro for dns is that it has been a part of the internet for a long time, has been well documented and is cross platform. The DDNS part is also widely in use and well supported.

DDNS certainly makes sense in situations where someone wants to run a FujiNet server from a non-static public IP.  No argument there.  That, however, would be best left as a function of the DDNS clinet running on their system, firewall, etc. to determine the URI of their particular site.

 

By the same token, though, this is something that should be an opt-in feature.  Not everyone is going to want to run a FujiNet site, and I would have significant privacy concerns over making what is essentially a centralised registration of the device's address something that happens automatically.

 

DNS also has its downsides.  One is propagation time (which is in the forefront of my mind right now as I'm sitting here waiting on three TXT records to propagate as I type).  Another is the technical barrier to using it: you and I may be familiar with it, but for J. Random Enduser, even an A record can be confusing.  Being able to enter a URI in a submission form and hit submit keeps the barrier to entry low.

38 minutes ago, SenorRossie said:

DNS is used for registration and storing of host records, added bonus: Global distributed storage.

While I'm not disagreeing with this idea in principle, how would a FujiNet record differ from any other?  An A / CNAME record with a TXT entry describing it?  Again, I may be missing something in all of this, but I'm not seeing how it would differ from how DNS is normally used.

38 minutes ago, SenorRossie said:

A second step could be axfr'ing (or ixfr if using slaves*) the zone to one or more central systems, checking the records and publishing them in a format easily accessible/parsable by the fujinet device.

 

* When using slaves to generate verified lists, an automated solution is feasable where an ixfr triggers regeneration of the published list.

Which makes sense, at least from the replication perspective.  A host could do a DDNS registration with tnfs.online (as an example), and appear in its zone file, which would then be globally-accessible.

 

This is all well and good, but there still needs to be a common method of locating and transferring / viewing resources on the network.  Note that I'm not opposed to DNS playing a role in that: it's something I'd considered necessary from the start.  I do have a concern about cache poisoning, but that's an out-in-the-weeds consideration for the moment.  DNSSEC can mitigate that (and other) concerns, but keeping the software stack simple and UX smooth are two of the main goals.

 

Let me think about this for a bit - I can see where a hybrid approach of the server handling DDNS registrations also generating the top-level gopherspace hierarchy based on DDNS might work.  It's a different approach to how I'd orginally thought to implement it, but that's OK because given how FujiNet and TNFS are expanding, I'm no longer convinced that my original ideas would work in the way that they would need to.

[SenorRossie]

1 hour ago, x=usr(1536) said:

3 hours ago, SenorRossie said:

The distinction between 'official' and user contributed hosts is entirely optional and could be as simple as using a different (sub-) domain.

What is the benefit of having this kind of separation?  Maybe I'm missing something in your intent behind the idea, but I'm not clear as to how having that separation would be necessary or desirable.

The intent is to have a 'trustworthy' domain maintained by some 'officials', i.e. developers of fujinet; and a 'user contributed' domain, where anyone can register that wants to be added to the list. Not a requirement per se, but I imagined a scenario where newly deployed fujinets get the official lists and a user is able to add one or more 'unofficial' domains where - for example - a user group sets up a domain for a select set of individuals, each running their own tnfs server(s).

For simplicity sake, let's start with one domain, document the setup and allow others to replicate it with their own domains.

 

1 hour ago, x=usr(1536) said:

Quote

A pro for dns is that it has been a part of the internet for a long time, has been well documented and is cross platform. The DDNS part is also widely in use and well supported.

DDNS certainly makes sense in situations where someone wants to run a FujiNet server from a non-static public IP.  No argument there.  That, however, would be best left as a function of the DDNS clinet running on their system, firewall, etc. to determine the URI of their particular site.

It could be a client running on their server/firewall and as most of these already are capable of creating a DDNS request, it allows for a relative easy implementation - Knowing that Joe Average may need some help in setting things up.

 

2 hours ago, x=usr(1536) said:

By the same token, though, this is something that should be an opt-in feature.  Not everyone is going to want to run a FujiNet site, and I would have significant privacy concerns over making what is essentially a centralised registration of the device's address something that happens automatically.

Hence the fact one needs to set up the client to register a site. All security concerns aside, it should always be a user choice to register a site.

 

2 hours ago, x=usr(1536) said:

DNS also has its downsides.  One is propagation time (which is in the forefront of my mind right now as I'm sitting here waiting on three TXT records to propagate as I type).  Another is the technical barrier to using it: you and I may be familiar with it, but for J. Random Enduser, even an A record can be confusing.  Being able to enter a URI in a submission form and hit submit keeps the barrier to entry low.

DNS has plenty of downsides and security concerns, but it is a proven technology and is widely used I do _not_ advocate on using a fujinet device to obtain 'the list' through DNS, so propagation time is a concern but should not be an issue per se. In my train of thought, the process would be something in the line of:

 - Client registers his site through DDNS, ie: usersite.fuji.net

   - The request is sent to (one of the) nameserver(s).

   - The registration of a host triggers a zone transfer on the (slave) nameserver(s)

 - On the slave servers the IXFR request triggers a regeneration the sitelist:

   - each host is checked for accessibility and if everything is oke, it is added to new sitelist file

 

The sitelist file can be a text-/json-/yaml-/etc. file and can be made accessible in a known location, ie:

 - a directory on a webserver

 - a directory on a tnfsserver

 - ...

The hostname and location of the file can be made part of the zone, i.e. an A record (or multiple for round robin redundancy) for the host and a TXT record for the location come to mind

 

2 hours ago, x=usr(1536) said:

Quote

DNS is used for registration and storing of host records, added bonus: Global distributed storage.

While I'm not disagreeing with this idea in principle, how would a FujiNet record differ from any other?  An A / CNAME record with a TXT entry describing it?  Again, I may be missing something in all of this, but I'm not seeing how it would differ from how DNS is normally used.

It wouldn't differ, except for the fact that with DDNS you have a method for the registration part.

 

That a lot of things need to be worked out is a given, as far as I am concerned. It seems easier to me to make use of available technology, instead of (re-)inventing something new. As far as I can tell, with the currently available protocols we should be able to make something work.

 

[+x=usr(1536)]

2 hours ago, SenorRossie said:

The intent is to have a 'trustworthy' domain maintained by some 'officials', i.e. developers of fujinet; and a 'user contributed' domain, where anyone can register that wants to be added to the list. Not a requirement per se, but I imagined a scenario where newly deployed fujinets get the official lists and a user is able to add one or more 'unofficial' domains where - for example - a user group sets up a domain for a select set of individuals, each running their own tnfs server(s).

Ah, ok - that makes sense.  In a way, this is already the case: see https://atari8bit.net/projects/software/fujinet-server-status/ for an example.  Granted, this isn't being done in an automated manner, but comes out to being functionally-equivalent in the end.

 

2 hours ago, SenorRossie said:

DNS has plenty of downsides and security concerns, but it is a proven technology and is widely used I do _not_ advocate on using a fujinet device to obtain 'the list' through DNS, so propagation time is a concern but should not be an issue per se. In my train of thought, the process would be something in the line of:

 - Client registers his site through DDNS, ie: usersite.fuji.net

   - The request is sent to (one of the) nameserver(s).

   - The registration of a host triggers a zone transfer on the (slave) nameserver(s)

 - On the slave servers the IXFR request triggers a regeneration the sitelist:

   - each host is checked for accessibility and if everything is oke, it is added to new sitelist file

From a technical standpoint, it's certainly feasible as far as the DNS side goes, and I am definitely in agreement re: not having every individual FujiNet grab the server list that way (or any other, for that matter)

 

The question then becomes how to make the server list human-readable and presentable to the client in as lightweight a way as possible.  This is a concern that isn't unique to the situation being proposed here, but it does highlight the need for a way - on the backend, not the FujiNet - to handle the consideration.

3 hours ago, SenorRossie said:

It wouldn't differ, except for the fact that with DDNS you have a method for the registration part.

True, but that also applies with LDAP.  Granted, LDAP is an idea I've more or less abandoned at this point, but agreed that credential storage and handling is a significant consideration in this context.  DDNS could help with that, definitely, but that comes back to the question of how to present the data in a way that's both useful and usable.

3 hours ago, SenorRossie said:

That a lot of things need to be worked out is a given, as far as I am concerned.

Oh, definitely.

3 hours ago, SenorRossie said:

It seems easier to me to make use of available technology, instead of (re-)inventing something new. As far as I can tell, with the currently available protocols we should be able to make something work.

This is the approach that was taken from the start: don't write anything you don't have to write, and have the FujiNet do as little as necessary of the processing.

[trent]

Any news on this? I'm looking to do something similar - much simpler, but I don't want to duplicate effort. I'm just thinking support could be added for gopher style directories to TNFS. This would allow site owners to provide browsable lists of TNFS hosts that a user could jump to. As a POC this would take the form of a specially formatted filename that the built-in TNFS client recognizes (i.e. <host>:<port>:<site name>), but I think the right way to do it is to add support for this to TNFS.

Edited January 27, 2022 by trent

[tschak909]

17 minutes ago, trent said:

Any news on this? I'm looking to do something similar - much simpler, but I don't want to duplicate effort. I'm just thinking support could be added for gopher style directories to TNFS. This would allow site owners to provide browsable lists of TNFS hosts that a user could jump to. As a POC this would take the form of a specially formatted filename that the built-in TNFS client recognizes (i.e. <host>:<port>:<site name>), but I think the right way to do it is to add support for this to TNFS.

Don't worry about stepping on toes. What works best is people building working prototypes, and showing them to each other. This way, rough consensus can be formed, and we can pull things together into something better.

 

-Thom

[trent]

Thanks, I'll give it a go then.

 

Looks like all the magic happens between these couple of files:

 

https://github.com/FujiNetWIFI/fujinet-config/blob/master/src/diskulator_select.c

https://github.com/FujiNetWIFI/fujinet-config/blob/master/src/diskulator_hosts.c

 

Most time consuming part is going to be setting up a build environment.

 

 

[tschak909]

16 minutes ago, trent said:

Thanks, I'll give it a go then.

 

Looks like all the magic happens between these couple of files:

 

https://github.com/FujiNetWIFI/fujinet-config/blob/master/src/diskulator_select.c

https://github.com/FujiNetWIFI/fujinet-config/blob/master/src/diskulator_hosts.c

 

Most time consuming part is going to be setting up a build environment.

 

 

If you want to see the other side of it, this is from the firmware side:

https://github.com/FujiNetWIFI/fujinet-platformio/blob/master/lib/device/sio/fuji.cpp

 

Config basically makes calls to the fuji.cpp device.

 

I'm glad you're digging in. Something I've said over and over: I have limited time for engineering. I make infinite time to teach.

 

-Thom

[trent]

Hmm. I was hoping for the POC I could avoid messing with the firmware, but I can see this will be more of a hack than I'd thought. When the host selection is made the current host list and the selection index is passed to fuji_sio_mount_host(), which means upon selection I'm going to have to have to fudge it by replacing one of the host slots ahead of that. For now I'm just going to hijack slot 8 and feel dirty about it. If that goes smoothly I'll dig a bit deeper for a better solution.

[tschak909]

1 minute ago, trent said:

Hmm. I was hoping for the POC I could avoid messing with the firmware, but I can see this will be more of a hack than I'd thought. When the host selection is made the current host list and the selection index is passed to fuji_sio_mount_host(), which means upon selection I'm going to have to have to fudge it by replacing one of the host slots ahead of that. For now I'm just going to hijack slot 8 and feel dirty about it. If that goes smoothly I'll dig a bit deeper for a better solution.

Go for it.  My role here is to answer questions.

 

-Thom

[trent]

First go attached. If a file starts with a '+', it assumes the rest is a fujinet tnfs hostname, writes it to slot 8 and opens it.

 

Next iteration I'll get rid of the slot 8 stuff - looks as if that is going to require messing with the firmware.

autorun.atr

[tschak909]

Do you have a run-through procedure?

-Thom

 

[trent]

There isn't much to it, other than access to a TNFS server is required.

 

1) Put the 'autorun.atr' file I attached in the previous post on your TNFS server.

2) Also on your TNFS server, create an empty file starting with a '+' and ending in the TNFS host you want to link to (e.g. "+fujinet.pl")

3) Boot up fujinet with the default config program

4) Navigate to the new 'autorun.atr', mount it on drive one and reboot (you will now be using the modified config program)

5) Navigate to the special file you created (e.g. "+fujinet.pl") and select it. This will cause you to jump to fujinet.pl, and if you look at your host list, #8 will be "fujinet.pl"

 

I'm working on a couple of additional changes that hide the magic prefix and show a little globe icon or some such in the file list, and doesn't hijack slot #8. I might have that ready tonight, if not then over the weekend.

Edited January 28, 2022 by trent

[tschak909]

Ah ok! very simple approach!

Anybody else have thoughts?

 

-Thom

 

[trent]

Yeah - there isn't much code changed. Whoever wrote/conceived of the config shell program did a fantastic job making a useful and intuitive interface and the code is very easy to understand. I've just for a while wished it had the capability to jump to different hosts like this. 

[tschak909]

Thank you so much. I did that work.

 

I've actually been writing the next version of it for the Coleco Adam version of FujiNet, and will back-port it to the Atari, you can see it here: 

https://github.com/FujiNetWIFI/fujinet-config-adam

 

Is it cleaner? I had gone overboard in the Atari config and overengineered some things.

 

-Thom

[tschak909]

@trent It would be great if you came by the discord, in the #meatballs channel, where there is a discussion happening over a distributed hash table approach to finding files and servers. 

Permanent invite code = https://discord.gg/7MfFTvD

[E474]

Off-the-wall suggestion, but could you use the HSC for a list of tnfs servers?

[tschak909]

*shrug* sure?

 

-Thom

 

[trent]

Thanks! I'll poke my head in later tonight. I'm tvi910. 

[trent]

I read through the #meatballs thread - much bigger idea than mine ?

 

I like that Fujinet is as simple as can be - plug it in, add a couple TNFS hosts, and you are off to the races. TNFS seems capable of delivering the sort of discovery functionality that is being discussed in this thread, and has the advantage that it is already part of the Fujinet ecosphere. The default TNFS host list can be seeded with key sites, and from there a user can jump off to other sites. Curating and organizing site lists would be up to TNFS operators, and the important ones would be baked into the firmware or referenced from one of the sites that is. 

 

I've updated the POC if anyone would like to try it out themselves:

 

* Adding 54.88.43.54 to your TNFS host list (don't add it as host #8, that slot is used by the demo)

* open 53.88.43.54 and download the 'autorun.atr' image, boot into it (this will not overwrite your config or firmware, you just load it like any other disk image)

* When Fujinet comes back up, open the 54.88.43.54 host again and browse to the "/sites" directory

* Sites will have an up arrow icon to the left. Select a site - the browser will open the site and it will be set as #8 in your TNFS host list.

 

You can add 'links' to your own tnfs site by creating files in the form "+<hostname>". These will appear as ordinary files to normal Fujinet users.

[mozzwald]

54 minutes ago, trent said:

I've updated the POC if anyone would like to try it out themselves:

 

* Adding 54.88.43.54 to your TNFS host list (don't add it as host #8, that slot is used by the demo)

I tried several times to boot the autorun.atr from that ip and it would just timeout so I used the python tnfs client to grab the atr and it's now hosted on fujinet.online tnfs as autotnfs.atr where I got it to boot successfully. I've also added a links dir to fujinet.online with some more tnfs servers.

 

Very cool!

 

Now we need that atr with the fast loader  

[_The Doctor__]

Maybe Fujinet would benefit from having 15 drive slots... since there are DOS's that support this, and then you could fill in upper slots with tnfs addresses if you like as well.

1-8 / I-O    15 total

said another way

A-O being   15 total...

like spartados does it... etc...

Edited January 29, 2022 by _The Doctor__

[trent]

15 hours ago, mozzwald said:

I tried several times to boot the autorun.atr from that ip and it would just timeout so I used the python tnfs client to grab the atr and it's now hosted on fujinet.online tnfs as autotnfs.atr where I got it to boot successfully. I've also added a links dir to fujinet.online with some more tnfs servers.

 

Very cool!

 

Now we need that atr with the fast loader  

My TNFS server is hosted on a well-worn AWS free tier instance so I guess mileage may vary. I tried from your site and it works well - thanks!

 

A proper implementation would not override the #8 host slot, but that will require some small changes to the firmware, which would make it a lot harder to try out the POC so I'll leave it as is for now. The link format could probably be more well thought out to make sure the format is unlikely to collide with real filenames.

 

There isn't much in the way of code changes, but I'm keeping them here temporarily: https://github.com/diggerbonk/fujinet-config/tree/file_links_poc

[trent]

On 1/27/2022 at 6:40 PM, tschak909 said:

Thank you so much. I did that work.

 

I've actually been writing the next version of it for the Coleco Adam version of FujiNet, and will back-port it to the Atari, you can see it here: 

https://github.com/FujiNetWIFI/fujinet-config-adam

 

Is it cleaner? I had gone overboard in the Atari config and overengineered some things.

 

-Thom

My comment on the code sprang from working with it - I had assumed that making the change I was thinking of was going to be difficult, I was pleasantly surprised to find things were well laid out and easy to understand. Looks like with the new code you are shooting to be platform agnostic which has added some indirection. Still looks straight forward though. It is interesting to see what you are expecting the "lowest common denominator" to be. I know those are hard decisions to make, they end up being long lived :-).