Jump to content
IGNORED

Directory Service for publicly-accessible #FujiNet sites


Recommended Posts

First and foremost: this is not a feature request.  Featureset identification and feasibility discussion, definitely, but *not* a feature request.

 

I've been kicking around the idea of a Directory Service for #FujiNet users.  The basic idea is that it would allow anyone with a #FujiNet to easily find publicly-accessible #FujiNet servers, as well as providing a way for people looking to run those servers an easy way to publish them as being available.

 

This would be a very lightweight protocol, with some similarities to DNS such as root servers containing a central list of server ip addresses and hostnames.  However, I can also see this including extended records (not totally dissimilar to a DNS TXT record) for each entry that describe things like the system type (physical, emulated via SIO2PC, Linux / Windows daemon, etc.), physical location (either using ccTLDs or FidoNET-style unique identifiers), and other info that may be worth providing (like a 40-character-or-less description of what the server offers).

 

Obviously, this would mean having to support the protocol on the #FujiNet side.  Implementing certificates may also be necessary if secure updates of hostnames, etc. on the root servers are considered necessary (which, IMHO, they should be), as well as UI changes to allow for configuring data sent to the root servers.

 

Again, just kicking the idea around for now.  Perhaps a P2P version of this may be less work, as an existing protocol (BitTorrent?) could be used or adapted as needed.

  • Like 5
Link to comment
Share on other sites

20 minutes ago, tschak909 said:

COOL! Build something. :) 

-Thom

OK, lemme chew it over :-D

 

I have banged on other folks' network code before, so do have a *little* experience in that department.  Time is the killer, though - right now I'm juggling work, school, and a career change.

 

Gonna think on it some more and see if there isn't something that could be adapted rather than starting from scratch if at all possible, but am open to suggestions ?

  • Like 1
Link to comment
Share on other sites

20 minutes ago, DjayBee said:

Why not use DNS and service records for this? Since @x=usr(1536) plans with some central "distribution point" anyway, this could as well be Thom's fujinet.online DNS zone.

 

This way you only need to do some DNS lookups.

I've given this consideration as well (actually, a combination of SRV and TXT records) and agree that it may work.  It's not out of the running, but may not be as flexible as using LDAP.

 

One of the things I'd like to see is the ability to tag a record type as being more than just the equivalent of a DNS A record.  In other words, being able to specify that it's a URL to a remote TNFS server, file type, located in a particular country, etc. within the structure of the entry itself brings up the possibility of having a lot of flexibility on the backend of how to display the data.

 

I'm not certain that DNS can do this as well as LDAP might be able to, given the structure of DNS records.  But I'm definitely not opposed to the idea, and if there's a way for it to be workable then I'm all ears.

 

FWIW, if it starts looking like this may need to be a blend of LDAP and DNS, it may be easier to just spin up an Azure instance and run AD since that's pretty much what AD is at its core.

Link to comment
Share on other sites

11 hours ago, x=usr(1536) said:

Thinking about this a bit more, LDAP might actually be a good way of handling it.

A quick search and I found this https://github.com/chrisridd/LDAPClient which is for esp32 Arduino. Would need some modification to make it work on PlatformIO.

 

17 minutes ago, x=usr(1536) said:

FWIW, if it starts looking like this may need to be a blend of LDAP and DNS, it may be easier to just spin up an Azure instance and run AD since that's pretty much what AD is at its core.

Whatever you decide to use, make sure it works cross platform. Something like OpenLDAP instead of Active Directory. Maybe they are compatible with each other, I don't know.

  • Like 1
Link to comment
Share on other sites

15 minutes ago, mozzwald said:

A quick search and I found this https://github.com/chrisridd/LDAPClient which is for esp32 Arduino. Would need some modification to make it work on PlatformIO.

Thanks for that.  Added to the list of things to look into.

15 minutes ago, mozzwald said:

Whatever you decide to use, make sure it works cross platform. Something like OpenLDAP instead of Active Directory. Maybe they are compatible with each other, I don't know.

LDAP can query AD, so that's not a problem.  Either way, anything I may do is going to be done under either macOS or Linux, so someone else would have to pick up the Windows side.

Link to comment
Share on other sites

Giving this more thought, it may be possible to implement this with minimal changes to the @FujiNet side of things.  tnfsd is where the effort should be concentrated.

 

If tnfsd can be built with the ability to query DNS / LDAP / whatever directory service, it can present that info to the #FujiNet in lieu of disk images, executables, etc.  This should also allow existing search functionality to work.

 

At that point, <insert directory service here> inbound to the root directory server would only be needed to register a given public tnfs instance.

 

That would rule out using the #FujiNet as a tnfs server, at least until tnfsd could be ported to it.  But that's a discussion for another day.

Link to comment
Share on other sites

OK... It appears as though this might be largely-doable by using the right software stack. I'll start building out a prototype system this weekend, time permitting.

 

Hardware will be a Raspberry Pi 3B+ running whatever current Raspbian (or Raspberry OS, or whatever they're calling it this week) is.  This is likely to move over to Debian under an x64 VM if things go well.  I'll need to do some cleanup on the VMware server to make that happen, which is something I really don't want to get into right now so am going with the RasPi.

 

Software stack:

The basic idea is that OpenLDAP runs on <instance> and provides the LDAP tree describing the public-facing tnfs servers.  FUSE lets that tree be presented as a standard mounted filesystem courtesy of ldapfuse.  Finally, tnfsd points to that LDAP mount for the server data it provides to the public.

 

Getting the four installed is a no-brainer; configuration may be interesting.  ldapfuse hasn't been updated in a long time, and there's no telling how well (if at all) it'll work with a current version of FUSE.

 

First goal is to get tnfsd to present the data to a client.  If that's successful, there needs to be a way for tnfsd to tell the #FujiNet that it's providing server addresses, not executables or media images; this is one change I can see being needed on the #FujiNet side.

 

Second goal is to be able to replicate the LDAP data from the local server to another.  This is to be tested locally first, but with the intention of ultimately making it something that happens across the Internet.  The idea is that this will ultimately allow for geographically-distributed root servers.

 

Final goal is to allow for individual tnfsd instances to register as public-facing with the root servers, thereby putting them in the public directory.  That whole end of things is very much TBD.

 

Any and all assistance with this is appreciated.  I'm perfectly happy to not be flying solo on this if there are better ideas out there :-D

Link to comment
Share on other sites

1 hour ago, x=usr(1536) said:

OK... It appears as though this might be largely-doable by using the right software stack. I'll start building out a prototype system this weekend, time permitting.

 

Hardware will be a Raspberry Pi 3B+ running whatever current Raspbian (or Raspberry OS, or whatever they're calling it this week) is.  This is likely to move over to Debian under an x64 VM if things go well.  I'll need to do some cleanup on the VMware server to make that happen, which is something I really don't want to get into right now so am going with the RasPi.

 

Software stack:

The basic idea is that OpenLDAP runs on <instance> and provides the LDAP tree describing the public-facing tnfs servers.  FUSE lets that tree be presented as a standard mounted filesystem courtesy of ldapfuse.  Finally, tnfsd points to that LDAP mount for the server data it provides to the public.

 

Getting the four installed is a no-brainer; configuration may be interesting.  ldapfuse hasn't been updated in a long time, and there's no telling how well (if at all) it'll work with a current version of FUSE.

 

First goal is to get tnfsd to present the data to a client.  If that's successful, there needs to be a way for tnfsd to tell the #FujiNet that it's providing server addresses, not executables or media images; this is one change I can see being needed on the #FujiNet side.

 

Second goal is to be able to replicate the LDAP data from the local server to another.  This is to be tested locally first, but with the intention of ultimately making it something that happens across the Internet.  The idea is that this will ultimately allow for geographically-distributed root servers.

 

Final goal is to allow for individual tnfsd instances to register as public-facing with the root servers, thereby putting them in the public directory.  That whole end of things is very much TBD.

 

Any and all assistance with this is appreciated.  I'm perfectly happy to not be flying solo on this if there are better ideas out there :-D

I will try to provide as much support as I can, to answer questions etc. My time is literally being spent on writing and rewriting N:.

 

-Thom

 

  • Like 1
Link to comment
Share on other sites

26 minutes ago, tschak909 said:

I will try to provide as much support as I can, to answer questions etc. My time is literally being spent on writing and rewriting N:.

Understood, and thanks.  Last thing I want to do is add to the pile either you or @mozzwald already have.

  • Like 1
Link to comment
Share on other sites

Still banging away at this, but progress is slow due to outside factors.

 

It's also become apparent that a Gopher-type interface with some form of Archie or Veronica for searching would be very useful once multiple TNFS sites are browsable from a directory.

 

No, I am not going to go all 'ooh, shiny!' on that and try to implement it :-D  Someone else can take that one on if they want.

Link to comment
Share on other sites

Progress has been made.  Here's where things stand at the moment:

 

The tnfs.online domain was registered today in order to give the public-facing servers somewhere to live under a common domain when the time comes.  It'll also make building & testing in an environment closer to what the final one will be easier.

 

The software stack (OpenLDAP / FUSE / ldapfuse / tnfsd) is in place and essentially working at a rudimentary level.  LDAP contains no data at the moment, largely because I'm still working out how to structure it; more on that below.  Barring a pressing need to change these components, I'd expect them to more or less be set in stone.

 

ldapfuse is being weird about mounting the LDAP tree.  It does it, but not in a way that makes it completely navigable, or with permissions that tnfsd is happy with.  This is the first time I've used ldapfuse, so am chalking this up to my unfamiliarity with it; it doesn't behave exactly like other FUSE filesystems do, so there is a learning curve.

 

Re: LDAP data structure: this is completely undecided at this point since I've been focussing on just getting software running.  However, there are certain criteria in terms of remote site capabilities that I'd like to list - for example, if a site allows uploads, set an 'incoming' flag.  Root servers get a 'root' flag, country names are flagged by TLD, etc.  The idea behind this is to make it easy to search by certain criteria, which in turn should make it easy to present the data to the tnfs client.

 

If anyone has ideas, suggestions, or wants to help, I'm 107% in favour and contributions are gratefully received.

Link to comment
Share on other sites

41 minutes ago, x=usr(1536) said:

Progress has been made.  Here's where things stand at the moment:

 

The tnfs.online domain was registered today in order to give the public-facing servers somewhere to live under a common domain when the time comes.  It'll also make building & testing in an environment closer to what the final one will be easier.

 

The software stack (OpenLDAP / FUSE / ldapfuse / tnfsd) is in place and essentially working at a rudimentary level.  LDAP contains no data at the moment, largely because I'm still working out how to structure it; more on that below.  Barring a pressing need to change these components, I'd expect them to more or less be set in stone.

 

ldapfuse is being weird about mounting the LDAP tree.  It does it, but not in a way that makes it completely navigable, or with permissions that tnfsd is happy with.  This is the first time I've used ldapfuse, so am chalking this up to my unfamiliarity with it; it doesn't behave exactly like other FUSE filesystems do, so there is a learning curve.

 

Re: LDAP data structure: this is completely undecided at this point since I've been focussing on just getting software running.  However, there are certain criteria in terms of remote site capabilities that I'd like to list - for example, if a site allows uploads, set an 'incoming' flag.  Root servers get a 'root' flag, country names are flagged by TLD, etc.  The idea behind this is to make it easy to search by certain criteria, which in turn should make it easy to present the data to the tnfs client.

 

If anyone has ideas, suggestions, or wants to help, I'm 107% in favour and contributions are gratefully received.

Need to find (or make) a C API for LDAP. openldap's libraries are a bit big to squooze onto an ESP32.

 

-Thom

 

  • Like 1
Link to comment
Share on other sites

28 minutes ago, tschak909 said:

Need to find (or make) a C API for LDAP. openldap's libraries are a bit big to squooze onto an ESP32.

 

-Thom

 

You may have just obliquely raised a UX point I've been wondering about for a while :-D

 

It seems to me that there are more than a couple of ways to present the server list to @FujiNet clients.  The idea that has been stuck in my head is that by default (meaning: as shipped when a #FujiNet is purchased) there's a 'master' entry for finding TNFS servers in one of the host slots.  Something like 'welcome.to.[fujinet || tnfs || etc.].online' or similar, which is doing a round-robin of the tnfs.online directory servers.  Selecting that gives the highest-level drill-down options, from which the user can get to the server they want.

 

Doing this would hopefully eliminate the need for building an ESP32 LDAP client, since everything is being presented over TNFS.  There may be tweaks needed to the TNFS client on the #FujiNet in order to process certain LDAP attributes presented via TNFS on its end, but I am aiming for as much as possible to be handled server-side in order to keep the changes on the client end as minimal as possible.

 

That approach does remove the possibility of every #FujiNet being able to also act as a TNFS server.  Not ideal, but there are also arguments in favour of keeping that functionality separate.  Totally separate discussion, though.

 

Obviously, it's all up in the air right now, so everyone's mileage may vary ;)  That's where my thoughts are at the moment, however.  Figured it might be helpful if I shared them.

Edited by x=usr(1536)
Link to comment
Share on other sites

1 minute ago, x=usr(1536) said:

You may have just obliquely raised a UX point I've been wondering about for a while :-D

 

It seems to me that there are more than a couple of ways to present the server list to @FujiNet clients.  The idea that has been stuck in my head is that by default (meaning: as shipped when a #FujiNet is purchased) there's a 'master' entry for finding TNFS servers in one of the host slots.  Something like 'welcome.to.[fujinet || tnfs || etc.].online' or similar, which is doing a round-robin of the tnfs.online directory servers.  Selecting that gives the highest-level drill-down options, from which the user can get to the server they want.

 

Doing this would hopefully eliminate the need for building an ESP32 LDAP client, since everything is being presented over TNFS.  There may be tweaks needed to the TNFS client on the #FujiNet in order to process certain LDAP attributes presented via TNFS on its end, but I am aiming for as much as possible to be handled server-side in order to keep the changes on the client end as minimal as possible.

 

Obviously, it's all up in the air right now, so everyone's mileage may vary ;)  That's where my thoughts are at the moment, though.  Figured it might be helpful if I shared them.

+1 for out of the box thinking. I like this.

-Thom

 

  • Like 1
Link to comment
Share on other sites

3 minutes ago, tschak909 said:

+1 for out of the box thinking. I like this.

-Thom

 

Thanks :)  ldapfuse really requires the kudos on this one, since it's the glue that could make all of this possible provided I can beat it into submission.

 

Bear in mind that I spent the last 15 years of my now-ended IT career as a manager at one level or another.  I'm nowhere near as technical as I used to be, but I've at least been able to remain a monkey that can occasionally make fire ;)

Link to comment
Share on other sites

  • 4 months later...
On 3/24/2021 at 5:38 AM, x=usr(1536) said:

FWIW, if it starts looking like this may need to be a blend of LDAP and DNS, it may be easier to just spin up an Azure instance and run AD since that's pretty much what AD is at its core.

You lost me at Azure!  AWS has a light weight directory service which mimics AD without the cost.  I do like the LDAP approach.  But I used to work at Netscape so I might be biased.

 

I’m no longer a practicing software engineer so take this with a block of salt but it seems to me a lambda function could serve as a listener for HTTP request from a FujiNet and then act as a real world to Atari translator so that all the Atari and or FujiNet would have to worry about is updating the display at x,y with a ATASCII characters received from

the lambda…

  • Like 2
Link to comment
Share on other sites

  • 2 weeks later...
8 hours ago, Kyle22 said:

That's nice, but Amazon could shut it down in an instant if [they] don't like how we're using it or what we say.

There has got to be a censorship free service for this.

 

 

So what? 

Then we move on to an Azure Function or a similar service in Tin Foil Cloud. 

  • Like 1
  • Haha 1
Link to comment
Share on other sites

4 hours ago, SenorRossie said:

Wouldn't a 'simple' ddns (RFC2136 style) client simplify things? If needed you could split it into official and user contributed content.

 

Just my $.02

Maybe.  I gave the idea some thought earlier on, and am not opposed to it in concept, but my position is that there needs to be no distinction between hosts as being official or otherwise.  The one possible exception of having DNS-style root servers that help to replicate directory data globally might be necessary, but, much like the Internet at large, I feel it's important to keep this as unmanaged as possible.

 

There's another consideration at play here: the FujiNet is well on its way to becoming truly multiplatform.  This is great news, since it means there'll be a common way to interconnect 8-bit machines.  However, with nearly 2000 FujiNets already out there, an LDAP-style directory might not be the best way to go about it.

 

As it grows, what I'm more and more convinced is needed is a common-ground protocol across architectures for communication by text and images.  Essentially, HTTP, but superlightweight.  Gopher is more like what I'm thinking of, along with a Veronica-style search engine (running on a submission basis rather than crawling) for the entire space.

 

Thinking about it, I wonder if the people at the Overbite Project might not be willing to assist with some of this.  Hmm.

Link to comment
Share on other sites

13 hours ago, Kyle22 said:

That's nice, but Amazon could shut it down in an instant if [they] don't like how we're using it or what we say.

Same applies to any service: you use someone else's environment, you play by their rules.

13 hours ago, Kyle22 said:

There has got to be a censorship free service for this.

Given that everything we're discussing is theoretical at this point and no threat of censorship exists as a result, this is pretty much putting the cart before the horse.  Granted, I'm not saying that this is something to not be concerned about - it is - but until it becomes a recognisable problem, I think we're safe to figure out the implementation details first, then worry about breaking a cloud host's TOS later.

  • Like 1
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Loading...
  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...