# Deploying without static IPs / hostnames **URL:** \ **Category:** General\ **Created:** [May 3, 2022, 7:54pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612 "2022-05-03T19:54:39Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![spiffytech](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/spiffytech/32/543_2.png) [@spiffytech](https://forum.yugabyte.com/u/spiffytech)\ **Post date:** [May 3, 2022, 7:54pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/1 "2022-05-03T19:54:39Z")
I’m trying to deploy Yugabyte to [Fly.io](http://Fly.io), which doesn’t have fixed instances, and has no way to address a specific instance (no static IPs, no fixed hostnames). I have a DNS entry that points to all of my Yugabyte master instances (`yb-masters.internal`), but that’s it. What do I need to enter for `--master_addresses`? I’ve tried entering `yb-masters.internal` but the masters can’t talk with each other. I also need to set `--rpc_bind_addresses` to IPv6 `::1` ([Fly.io](http://Fly.io)’s internal networking is IPv6), and I can’t tell if I’m getting the syntax for that right until I figure out how to get the masters to talk to each other. Using this command: ```auto /home/yugabyte/bin/yb-master --fs_data_dirs=/data/master \ --master_addresses=yb-masters.internal:7100 \ --replication_factor=3 --logtostderr ``` I get the error: ```auto Error getting permanent uuid from config peer [yb-masters.internal:7100]: Network error (yb/util/net/socket.cc:540): recvmsg error: Connection refused (system error 111) ``` If I instead use this command: ```auto /home/yugabyte/bin/yb-master --fs_data_dirs=/data/master \ --master_addresses=yb-masters.internal:7100 \ --rpc_bind_addresses=localhost6:7100 \ --replication_factor=3 --logtostderr ``` I get this error: ```auto Unable to init master catalog manager: Illegal state (yb/master/catalog_manager.cc:1644): Unable to initialize catalog manager: Failed to initialize sys tables async: None of the local addresses are present in master_addresses yb-masters.internal:7100 ``` ---
**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\ **Post date:** [May 4, 2022, 6:33am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/2 "2022-05-04T06:33:07Z")
Hi @spiffytech > [@spiffytech](#): > > I have a DNS entry that points to all of my Yugabyte master instances ( `yb-masters.internal` ), but that’s it. There really isn’t at least a fixed hostname for servers? How are you supposed to differentiate between them (say, in your custom code)? Can you ask the support staff? Maybe by using a script that queries the DNS on startup, getting all the entries, and using those to start the yb-master/tserver? ---
**Author:** ![FranckPachot](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/franckpachot/32/353_2.png) [@FranckPachot](https://forum.yugabyte.com/u/FranckPachot)\ **Post date:** [May 4, 2022, 7:31am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/3 "2022-05-04T07:31:34Z")
Hi @spiffytech how did you deploy on [fly.io](http://fly.io)? If there is one DNS entry per application, I guess each yb-master should be one application. Or add some logic like they do with PostgreSQL ([GitHub - fly-apps/postgres-ha: Postgres + Stolon for HA clusters as Fly apps.](https://github.com/fly-apps/postgres-ha)) I’ll try to get info from [fly.io](http://fly.io) people. I didn’t test YugabyteDB on [fly.io](http://fly.io) but that’s something I wanted to do Franck. ---
**Author:** ![spiffytech](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/spiffytech/32/543_2.png) [@spiffytech](https://forum.yugabyte.com/u/spiffytech)\ **Post date:** [May 4, 2022, 4:22pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/4 "2022-05-04T16:22:45Z")
> [@dorian\_yugabyte](#): > > There really isn’t at least a fixed hostname for servers? How are you supposed to differentiate between them (say, in your custom code)? Can you ask the support staff? Maybe by using a script that queries the DNS on startup, getting all the entries, and using those to start the yb-master/tserver? [Fly.io](http://Fly.io) doesn’t have a notion of “servers” - it’s just ephemeral containers, possibly with storage attached. (They encourage the “cattle, not pets” mantra so you can’t really get a persistent identity for server01, server02, etc. besides whatever’s stored on disk) There’s a DNS entry that will give me the IPs of _running_ instances, but it won’t have 3 IPs until all 3 yb-masters have begun booting, introducing a chicken-and-egg problem since I need the 3 IPs to boot the masters. I could be hacky and `sleep` on boot until all 3 instances had launched and been issued IPs. I don’t think Fly guarantees that instance IPs are static; if I redeploy then my instances could get different IPs. > [@FranckPachot](#): > > Hi @spiffytech how did you deploy on [fly.io](http://fly.io)? If there is one DNS entry per application, I guess each yb-master should be one application. Or add some logic like they do with PostgreSQL ([GitHub - fly-apps/postgres-ha: Postgres + Stolon for HA clusters as Fly apps.](https://github.com/fly-apps/postgres-ha)) > I’ll try to get info from [fly.io](http://fly.io) people. I didn’t test YugabyteDB on [fly.io](http://fly.io) but that’s something I wanted to do > Franck. I made a minimal Dockerfile `FROM yugabytedb/yugabyte`, overriding the `ENTRYPOINT` with the above commands. I created three instances of the app, each with at persistent volume. I’ll check out your suggestions. ---
**Author:** ![karthik](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/karthik/32/13_2.png) [@karthik](https://forum.yugabyte.com/u/karthik)\ **Post date:** [May 5, 2022, 3:44am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/5 "2022-05-05T03:44:07Z")
Hi @spiffytech, Thank you for trying YB! Would love to get it working with [fly.io](http://fly.io), I am adding some more folks here. - @sanketh @arnav - could you folks pls help here ---
**Author:** ![FranckPachot](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/franckpachot/32/353_2.png) [@FranckPachot](https://forum.yugabyte.com/u/FranckPachot)\ **Post date:** [May 5, 2022, 5:25pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/6 "2022-05-05T17:25:43Z")
@spiffytech I don’t think you can use a DNS resolving to multiple addresses like yb-masters.internal I got it working by resolving the addresses like this once scaled to 3 nodes: ```auto master_addresses=$(nslookup $FLY_APP_NAME.internal | grep -A1 "$FLY_APP_NAME.internal" | awk '/^Address:/{print "["$2"]:7100"}' | paste -sd,) set | grep ^master_addresses` ``` Here is what I’ve done to deploy to 3 regions: > Take it as a draft, was my first trial of [fly.io](http://fly.io) ---
**Author:** ![spiffytech](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/spiffytech/32/543_2.png) [@spiffytech](https://forum.yugabyte.com/u/spiffytech)\ **Post date:** [May 5, 2022, 6:38pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/7 "2022-05-05T18:38:06Z")
Excellent, I’ll start working from there. Thanks! Does Yugabyte assume the master/tserver IPs stay the same? I.e., if a master rebooted and came back on a different IP than before, would that cause problems? ---
**Author:** ![FranckPachot](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/franckpachot/32/353_2.png) [@FranckPachot](https://forum.yugabyte.com/u/FranckPachot)\ **Post date:** [May 5, 2022, 9:16pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/8 "2022-05-05T21:16:58Z")
Yes it would cause problems and require some `yb-admin` commands. [but see Karthik’s answer - not a problem id DNS name is used. So for [fly.io](http://fly.io), AFAIK, a restart keeps the IP and name of the VM, but a scale-down-scale-up changes both] What I think would be the most stable deployment: - define 3 yb-master ad 3 apps so that they have their host name, and then you can refer to them as `--master_addresses=yb-master-1.internal:7100,yb-master-2.internal:7100,yb-master-3.internal:7100` - define yb-tserver apps with one per region. So that you don’t rely on the guess that [fly.io](http://fly.io) will distribute them correctly. And those can then be scaled. But at least, you are sure that you assign 3 sets of master and servers to 3 regions. The only thing you have to take care then is scale each set of tserver to the same size. And don’t forget to define the placement when starting them (like `--placement_zone="${FLY_REGION}"`) Because for the masters, you don’t want to scale them. Having 3 for a 3 regions cluster is ok. You want elasticity for tservers, but being sure that YugabyteDB knows in which region they are, to be resilient to a region failure ---
**Author:** ![karthik](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/karthik/32/13_2.png) [@karthik](https://forum.yugabyte.com/u/karthik)\ **Post date:** [May 6, 2022, 3:52am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/9 "2022-05-06T03:52:44Z")
> [@spiffytech](#): > > Does Yugabyte assume the master/tserver IPs stay the same? I.e., if a master rebooted and came back on a different IP than before, would that cause problems? @spiffytech - no, the IPs can change (this is the way k8s works). In this case, you would need to use the server dns names rather than the ip address, and YugabyteDB will auto-refresh the ip. The identities of the various servers are stored as internal uuid’s, and the ip address becomes a dynamic attribute in this case (and the hostname a static attribute) - whereas normally, the ip address is treated as a static attribute. ---
**Author:** ![dorian\_yugabyte](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/dorian_yugabyte/32/206_2.png) [@dorian\_yugabyte](https://forum.yugabyte.com/u/dorian_yugabyte)\ **Post date:** [May 6, 2022, 7:19am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/10 "2022-05-06T07:19:21Z")
> [@spiffytech](#): > > [Fly.io](http://Fly.io) doesn’t have a notion of “servers” - it’s just ephemeral containers, possibly with storage attached. (They encourage the “cattle, not pets” mantra so you can’t really get a persistent identity for server01, server02, etc. besides whatever’s stored on disk) Note that “cattle” doesn’t work for stateful apps. Even when you’re Google. Even some databases that split compute & storage into separate tiers, the storage tier is “pet” and the compute tier is semi “pet/cattle”. ---
**Author:** ![spiffytech](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/spiffytech/32/543_2.png) [@spiffytech](https://forum.yugabyte.com/u/spiffytech)\ **Post date:** [May 6, 2022, 6:36pm UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/11 "2022-05-06T18:36:40Z")
I [brought this up](https://community.fly.io/t/can-an-instance-have-a-persistent-network-identity/5045/8) in the [Fly.io](http://Fly.io) forum as well, just to be sure I wasn’t missing anything. Fly _does_ assign a permanent private IP that gets paired with each storage volume, I just don’t think that’s documented anywhere. And volumes are locked to the region they’re created in, so it sounds like the answer is spin up dummy containers so I can enumerate their private IPs, then spin up Yugabyte masters configured with those IPs. Thanks for helping me work through this! ---
**Author:** ![FranckPachot](https://yyz1.discourse-cdn.com/flex027/user_avatar/forum.yugabyte.com/franckpachot/32/353_2.png) [@FranckPachot](https://forum.yugabyte.com/u/FranckPachot)\ **Post date:** [May 7, 2022, 5:52am UTC](https://forum.yugabyte.com/t/deploying-without-static-ips-hostnames/1612/12 "2022-05-07T05:52:16Z")
Awesome. As the volume creates the node identity this is perfect solution. Create 3 volumes, get the private IPs, then deploy masters with this list and scale to 3. Then create volumes for data, deploy and scale tservers with this list of masters. For single region you could also use yugabyte which does the admin of adding to the list for each. Just needs to check if it is the first one or get one of the other IP to join. Could be possible from nslookup. Thanks a lot. I’ll also test and document this for the community. Your feedback is great help