WEBVTT 1 00:01:03.900 --> 00:01:05.459 Ingolf Kuss: Oh, Shelly. 2 00:01:09.580 --> 00:01:10.530 Shelley Doljack: Hello! 3 00:01:15.260 --> 00:01:17.980 Shelley Doljack: I thought I wasn't gonna make it today, but… 4 00:01:18.580 --> 00:01:22.710 Ingolf Kuss: Oh, I've seen you in the TC already, so… 5 00:01:23.310 --> 00:01:24.050 Shelley Doljack: PM. 6 00:01:24.460 --> 00:01:31.100 Shelley Doljack: My husband's having surgery right now, so… Yeah, I'll just, like… 7 00:01:31.900 --> 00:01:37.179 Shelley Doljack: It's just, it's, you know, an outpatient thing, it's not major, so probably… 8 00:01:37.540 --> 00:01:39.110 Ingolf Kuss: May I just told us, but so… 9 00:01:39.110 --> 00:01:42.710 Shelley Doljack: Yeah, good luck. I'm just here to lurk right now. 10 00:01:44.620 --> 00:01:45.829 Ingolf Kuss: You hear an alert? 11 00:01:46.180 --> 00:01:47.140 Shelley Doljack: lurk. 12 00:01:47.310 --> 00:01:51.540 Shelley Doljack: Yeah, not necessarily participate. 13 00:01:52.210 --> 00:01:53.110 Ingolf Kuss: Okay 14 00:01:56.840 --> 00:01:58.879 Ingolf Kuss: Hello, Josh. Hello, Florian. 15 00:02:01.660 --> 00:02:02.380 josh greben: Hello. 16 00:02:13.280 --> 00:02:15.929 Ingolf Kuss: Jason cannot attend. 17 00:02:17.690 --> 00:02:19.620 Ingolf Kuss: Quite unfortunate. 18 00:02:21.250 --> 00:02:24.299 Ingolf Kuss: We wanted to discuss the, both con… 19 00:02:25.860 --> 00:02:32.600 Ingolf Kuss: possible presentation topics. I thought Jason would certainly have an idea. 20 00:02:33.340 --> 00:02:35.500 Ingolf Kuss: And we should have him in that discussion. 21 00:02:36.320 --> 00:02:44.600 Ingolf Kuss: So, maybe… I have heard that the deadline has been extended to April 15? 22 00:02:45.330 --> 00:02:48.639 Ingolf Kuss: So, we don't have to do that today. 23 00:02:53.430 --> 00:02:57.870 Ingolf Kuss: Margot, I think that's someone from the Bavarian. 24 00:02:57.870 --> 00:02:59.109 Florian Kreft (LRZ): Oh, yes. 25 00:02:59.110 --> 00:03:00.000 Ingolf Kuss: Network. 26 00:03:00.000 --> 00:03:11.810 Florian Kreft (LRZ): Maharaj is here, so… Just today, he managed to get the first Eureka… prototype… Thing… Installed, so… 27 00:03:11.930 --> 00:03:13.879 Florian Kreft (LRZ): That's, why I know… 28 00:03:14.060 --> 00:03:19.679 Florian Kreft (LRZ): Suggested that he… it might be valuable for him to attend these sessions as well. 29 00:03:20.850 --> 00:03:22.470 Ingolf Kuss: No, sure. 30 00:03:22.470 --> 00:03:29.920 Florian Kreft (LRZ): problems himself, or, like, anything else is discussed that might be relevant? 31 00:03:32.450 --> 00:03:37.459 Ingolf Kuss: Oh, okay. Yeah, do you do… you don't have to turn on your camera. 32 00:03:37.800 --> 00:03:39.500 Ingolf Kuss: Welcome, Mahat. 33 00:03:40.300 --> 00:03:41.209 Mahrad Zoonematkermani: Thank you very much. 34 00:03:41.210 --> 00:03:42.270 Ingolf Kuss: you're here. 35 00:03:42.730 --> 00:03:43.450 Mahrad Zoonematkermani: Thank you. 36 00:03:43.850 --> 00:03:44.980 Mahrad Zoonematkermani: Thanks for that. 37 00:03:45.240 --> 00:03:52.719 Ingolf Kuss: Oh… What was the success message? He installed a prototype of. 38 00:03:52.720 --> 00:03:53.530 Florian Kreft (LRZ): Yeah, so… 39 00:03:53.530 --> 00:03:53.870 Ingolf Kuss: What? 40 00:03:53.870 --> 00:04:03.700 Florian Kreft (LRZ): We, or he, rather, has been working on getting all the infrastructure stuff installed in a way that… 41 00:04:03.980 --> 00:04:21.089 Florian Kreft (LRZ): in cases where it's possible already represents, like, somewhat of an, of a state where we could use this in production, there are still quite a few issues we have to fix after that. The first benchmark was not to actually 42 00:04:21.089 --> 00:04:25.070 Florian Kreft (LRZ): Like, have all the infrastructure things installed. 43 00:04:25.070 --> 00:04:40.149 Florian Kreft (LRZ): have all the, like, applications, posted correctly, entitled the, a tenant, and then have a front-end actually working, including all of the dependencies we need. And we, we, I mean… 44 00:04:40.150 --> 00:04:57.550 Florian Kreft (LRZ): if we want to discuss in detail at some point the things Margot found that are definitely not okay yet for, like, productive use, that might be valuable at some point, but like I said, just today was the first, like, front-end build that can connect to Eureka. 45 00:04:58.210 --> 00:04:58.900 Florian Kreft (LRZ): Oh. 46 00:04:59.900 --> 00:05:03.599 Ingolf Kuss: The first front-end build that can connect to Eureka. 47 00:05:03.600 --> 00:05:05.259 Florian Kreft (LRZ): So that you can actually log in. 48 00:05:05.260 --> 00:05:07.449 Ingolf Kuss: It means you already have a backend. 49 00:05:08.030 --> 00:05:09.179 Mahrad Zoonematkermani: Oh, yeah, absolutely. 50 00:05:09.180 --> 00:05:09.840 Florian Kreft (LRZ): Gale. 51 00:05:10.920 --> 00:05:16.050 Florian Kreft (LRZ): I mean, it was just the first benchmark. Have an actual Stripes where you can log in. 52 00:05:16.630 --> 00:05:25.470 Shelley Doljack: Yeah, I'd be curious to know what, you think is not okay for going to production, because we're going to production this Saturday. 53 00:05:25.470 --> 00:05:25.810 Ingolf Kuss: Whoa. 54 00:05:25.810 --> 00:05:28.910 Mahrad Zoonematkermani: Oh, God. Best of luck. 55 00:05:29.520 --> 00:05:32.599 Mahrad Zoonematkermani: And thank you, Sheila, your guides helped a lot. 56 00:05:33.670 --> 00:05:34.210 Ingolf Kuss: Oh. 57 00:05:34.790 --> 00:05:35.520 Mahrad Zoonematkermani: Aww. 58 00:05:36.420 --> 00:05:37.230 Mahrad Zoonematkermani: So… 59 00:05:37.230 --> 00:05:39.150 Ingolf Kuss: Do you want to report on this? 60 00:05:39.580 --> 00:05:39.970 Mahrad Zoonematkermani: for us. 61 00:05:39.970 --> 00:05:43.460 Ingolf Kuss: I want to show you my issues later, but we can… 62 00:05:43.780 --> 00:05:45.010 Mahrad Zoonematkermani: Hmm… That was your… 63 00:05:45.010 --> 00:05:45.520 Ingolf Kuss: I mean, you're… 64 00:05:45.520 --> 00:05:47.370 Mahrad Zoonematkermani: I can start with yours, because. 65 00:05:47.370 --> 00:05:47.730 Ingolf Kuss: Okay. 66 00:05:47.730 --> 00:05:50.759 Mahrad Zoonematkermani: I haven't put my thoughts together yet. 67 00:05:51.250 --> 00:05:54.030 Mahrad Zoonematkermani: that, in the meantime, on the side, I can put together. 68 00:05:54.030 --> 00:06:18.070 Florian Kreft (LRZ): Yeah, sorry, I put Marat on the spot, so it's just, like, that came to mind from our discussions, there were still some open things in our environment that might be helpful to know for you to… maybe it's not relevant in your environment at all, but it might make sense to discuss them, if possible, yeah, but we have not prepared anything, so… 69 00:06:18.320 --> 00:06:24.290 Florian Kreft (LRZ): Marat, if you can gather your thoughts, we can do that, but otherwise… 70 00:06:24.950 --> 00:06:25.590 Florian Kreft (LRZ): Ugh. 71 00:06:25.930 --> 00:06:27.219 Mahrad Zoonematkermani: Can we have some hearts? 72 00:06:27.220 --> 00:06:28.170 Ingolf Kuss: Who knows. 73 00:06:28.490 --> 00:06:29.570 Mahrad Zoonematkermani: Sorry, go on. 74 00:06:29.570 --> 00:06:35.740 Ingolf Kuss: Maybe we could make a short introduction, since we have a new… member. 75 00:06:36.620 --> 00:06:39.659 minus: Hi, I'm not new, but I maybe changed my. 76 00:06:39.660 --> 00:06:42.420 Ingolf Kuss: But Marat is new, he doesn't know us, so maybe… 77 00:06:42.420 --> 00:06:43.610 minus: Okay. 78 00:06:43.610 --> 00:06:46.809 Ingolf Kuss: maybe two sentences. I'm Ingov, of course, from… 79 00:06:47.340 --> 00:06:52.610 Ingolf Kuss: HPZ, North Roy and West Australian Library Service Center, or so. 80 00:06:53.390 --> 00:06:58.040 Ingolf Kuss: And, goh, what? 81 00:06:58.150 --> 00:07:02.830 Ingolf Kuss: I've been a long-time convener for this group, but my… 82 00:07:03.610 --> 00:07:07.830 Ingolf Kuss: Deployment in Eureka is not production ready, and, 83 00:07:08.420 --> 00:07:13.390 Ingolf Kuss: Yeah, doing that on my own with 3 servers, so you will, see. 84 00:07:14.940 --> 00:07:17.239 Ingolf Kuss: Maybe, who wants to be the next? 85 00:07:17.920 --> 00:07:19.320 Ingolf Kuss: Maybe Magnus? 86 00:07:19.530 --> 00:07:20.990 Ingolf Kuss: You're out on a walk? 87 00:07:21.910 --> 00:07:23.620 minus: Yeah, I'm going home. 88 00:07:23.930 --> 00:07:32.279 minus: Oh, I'm called Minus, I'm a system administrator at Lin Shopping's University in Sweden. 89 00:07:32.440 --> 00:07:37.650 minus: We are upgrading to Ramson's in a month. 90 00:07:39.440 --> 00:07:47.240 minus: And we have run folios since… 5 releases ago, I think? 91 00:07:49.520 --> 00:07:51.079 Ingolf Kuss: 5 releases. 92 00:07:51.670 --> 00:07:52.210 minus: Different. 93 00:07:54.370 --> 00:07:56.380 Ingolf Kuss: Okay, Josh, maybe you? 94 00:07:57.060 --> 00:08:00.610 josh greben: Yeah, hi, Josh Groban from Stanford University, Systems Programmer. 95 00:08:03.000 --> 00:08:04.000 Ingolf Kuss: Okay. 96 00:08:04.710 --> 00:08:22.480 Shelley Doljack: I'm Shelly, I work with Josh at Stanford University. We've been hosting Folio on Kubernetes since 2023, as well, and like I said, we're gonna be going to Eureka Sunflower this Saturday. 97 00:08:23.420 --> 00:08:24.210 Ingolf Kuss: Whoa. 98 00:08:24.420 --> 00:08:25.050 Shelley Doljack: Yep. 99 00:08:25.800 --> 00:08:26.920 Mahrad Zoonematkermani: Fingers crossed. 100 00:08:27.460 --> 00:08:30.930 Ingolf Kuss: But you're going to, so you have everything ready, you have a… 101 00:08:31.960 --> 00:08:32.870 Shelley Doljack: It's already… 102 00:08:32.870 --> 00:08:36.810 Ingolf Kuss: And you just need to turn the switch over, or… 103 00:08:36.980 --> 00:08:37.799 Ingolf Kuss: No. 104 00:08:37.809 --> 00:08:39.069 Shelley Doljack: I mean, we gotta… 105 00:08:39.659 --> 00:08:57.239 Shelley Doljack: there's… we gotta upgrade, still, like, our prod, stuff, but we got, kind of, like, the pre-install stuff done. You know, now that all that's left is, like, you know, bring down OCopy, uninstall modules. 106 00:08:57.469 --> 00:09:00.109 Shelley Doljack: Put new ones out. Entitle. 107 00:09:00.610 --> 00:09:02.580 Ingolf Kuss: What about the data migration? 108 00:09:03.030 --> 00:09:09.260 Ingolf Kuss: Your tenant's data. Where are your tenant's data? It's in the… Copy database. 109 00:09:09.570 --> 00:09:11.570 Ingolf Kuss: Do we use the same database? 110 00:09:13.710 --> 00:09:14.690 Ingolf Kuss: Oh, okay. 111 00:09:14.690 --> 00:09:23.559 Florian Kreft (LRZ): Yeah, I mean, the module data should be the same in most cases. The one thing I'm really curious about is, like, the entitlement permission stuff. 112 00:09:23.960 --> 00:09:27.100 Florian Kreft (LRZ): What kind of migrations are necessary, because… 113 00:09:27.100 --> 00:09:27.950 Mahrad Zoonematkermani: That's right. 114 00:09:28.670 --> 00:09:34.540 Mahrad Zoonematkermani: I think there's a migration path for it. I haven't tried it, but I'm sure Shelly and Josh have. 115 00:09:34.740 --> 00:09:35.300 Florian Kreft (LRZ): Nope. 116 00:09:36.430 --> 00:09:39.920 Florian Kreft (LRZ): Have you run into any issues with that? Or. 117 00:09:40.730 --> 00:09:45.460 Shelley Doljack: Are you talking about the roles? 118 00:09:45.830 --> 00:09:47.050 Shelley Doljack: Migration. 119 00:09:47.180 --> 00:09:56.039 Shelley Doljack: Yup. So… Mod Roles KeyCloak provides an API endpoint to migrate permissions to roles and capabilities. 120 00:09:56.040 --> 00:09:57.000 Florian Kreft (LRZ): Okay. 121 00:09:57.000 --> 00:10:02.019 Shelley Doljack: And, when we first ran it, we didn't like it. 122 00:10:02.700 --> 00:10:10.830 Shelley Doljack: And I think at the time when we first ran it, we were not aware of the folio user permission mapper Python tool. 123 00:10:11.300 --> 00:10:16.920 Shelley Doljack: But now there's this Folio User Permission Mapper Python tool. And… 124 00:10:17.030 --> 00:10:29.679 Shelley Doljack: in, one of the early adopters meetings, Craig McNally said that, both of those, it… they designed them both to be run at the same time, so you would 125 00:10:29.760 --> 00:10:38.669 Shelley Doljack: use the modules KeyCloak API migrations endpoint, and then you would use the Python tool to, 126 00:10:39.410 --> 00:10:44.560 Shelley Doljack: There… at the end, there's, like, some cleanup of the stuff that the… 127 00:10:44.910 --> 00:10:56.150 Shelley Doljack: mod roles KeyCloat creates, it creates a bunch of roles that don't have display names that are friendly. They're UUIDs, the display names, and then… 128 00:10:56.320 --> 00:11:03.429 Shelley Doljack: I don't know, there's… so… We decided just to run the Python tool because it worked pretty well. 129 00:11:03.890 --> 00:11:10.480 Shelley Doljack: And it worked well because… We created… 130 00:11:10.680 --> 00:11:27.959 Shelley Doljack: permission sets in our system that are based around functional roles already. Like, when we first migrated to Folio, we were, like, had a group and decided how we're going to do these permissions, and so we have, like. 131 00:11:28.400 --> 00:11:33.730 Shelley Doljack: permission sets that are kind of, like, based around functional roles in the library. So those kind of, like. 132 00:11:34.380 --> 00:11:41.380 Shelley Doljack: mapped really nicely to roles in Eureka already for us. So… 133 00:11:42.440 --> 00:11:54.479 Shelley Doljack: Some… one of the issues, though, that I think is not… I put it in the Folio Eureka support channel. I don't really like what they decided there, but, 134 00:11:54.770 --> 00:12:10.890 Shelley Doljack: In a copy-based environment, if you have the permission to assign users permissions, you end up getting the capabilities to create roles and edit roles. 135 00:12:11.530 --> 00:12:30.380 Shelley Doljack: Which I think is a little bit different, because in Eureka, you have the ability to create roles and edit roles, but then there's also the capability to assign users to roles, which is in a separate capability set from the whole, like, UI authorization roles thing. 136 00:12:30.830 --> 00:12:39.299 Shelley Doljack: So, we… after we migrate, we're gonna have to take away some… some stuff from people. 137 00:12:39.460 --> 00:12:44.279 Shelley Doljack: So that they don't have more things than they should, but… 138 00:12:44.510 --> 00:12:54.019 Shelley Doljack: And then the other issue is that the authorization roles and authorization policies is not fully implemented with 139 00:12:54.100 --> 00:13:09.179 Shelley Doljack: the Stripes UI. So, like, if you go in as an administrative user that has all capabilities, you log in via Stripes, and you try to create an authorization policy. 140 00:13:09.370 --> 00:13:23.159 Shelley Doljack: nothing happens in the back end. It's broken. There's no connection between the two. You do have permissions to do a post, you know, if you wanted to do a curl and create an authorization policy, but Craig said that that stuff is kind of like… 141 00:13:23.760 --> 00:13:27.539 Shelley Doljack: not fully implemented in Sunflower, so there is some. 142 00:13:28.240 --> 00:13:30.980 Shelley Doljack: And, and going back and testing, like. 143 00:13:31.310 --> 00:13:35.839 Shelley Doljack: The capability sets and trying to figure out, like. 144 00:13:36.370 --> 00:13:43.580 Shelley Doljack: What is the right thing for allowing people to view capabilities, but not edit and… 145 00:13:43.940 --> 00:13:49.790 Shelley Doljack: delete and create roles and things like that. It didn't seem to… 146 00:13:50.490 --> 00:13:55.150 Shelley Doljack: I got, confounding, results. 147 00:13:55.280 --> 00:14:02.459 Shelley Doljack: So, I think it's just kind of buggy there. Maybe they fixed it in CSP5? 148 00:14:03.300 --> 00:14:06.420 Shelley Doljack: For Sunflower, we're gonna go with CSP4. 149 00:14:06.660 --> 00:14:07.610 Florian Kreft (LRZ): Okay. 150 00:14:07.610 --> 00:14:13.789 Shelley Doljack: Because that's what we've had a successful, smooth entitlement on our stage environment with. 151 00:14:14.280 --> 00:14:14.680 Florian Kreft (LRZ): Yep. 152 00:14:14.680 --> 00:14:23.530 Shelley Doljack: And then shortly after, we're gonna try to go to CSP5, I think, because… Of some metadata thing. 153 00:14:23.910 --> 00:14:29.249 Shelley Doljack: That is broken in CSV4, but yeah, that's my report. 154 00:14:29.770 --> 00:14:31.230 Florian Kreft (LRZ): Yeah, awesome. 155 00:14:31.990 --> 00:14:43.599 Florian Kreft (LRZ): So that's the main thing that was on my mind, but I've obviously not… so we are far from testing migration… migrations, I think. Well, maybe not too far, but… 156 00:14:43.770 --> 00:14:47.750 Florian Kreft (LRZ): We have not talked about it yet, 157 00:14:48.130 --> 00:14:52.570 Florian Kreft (LRZ): Were there any other data migrations needed? 158 00:14:52.990 --> 00:14:55.070 Florian Kreft (LRZ): Apart from roles? 159 00:14:55.580 --> 00:14:57.500 Florian Kreft (LRZ): No, right? Just… 160 00:14:57.500 --> 00:14:59.079 Shelley Doljack: the, the users? 161 00:14:59.220 --> 00:14:59.860 Florian Kreft (LRZ): Yeah? 162 00:15:00.140 --> 00:15:02.160 Shelley Doljack: You have to migrate your users. 163 00:15:02.380 --> 00:15:12.799 Shelley Doljack: So all… so Mod Users KeyCloak has an endpoint for migrating your users, and it takes all the users that have permissions in a copy. 164 00:15:13.130 --> 00:15:15.489 Shelley Doljack: And then it puts them in Key Cloak. 165 00:15:15.750 --> 00:15:19.720 Shelley Doljack: It creates the user records in KeyCloak, but also… 166 00:15:20.260 --> 00:15:23.740 Shelley Doljack: in the Folio database, some stuff. 167 00:15:25.680 --> 00:15:30.230 Shelley Doljack: We didn't run into any issues around that. 168 00:15:30.230 --> 00:15:33.309 Florian Kreft (LRZ): And you can run that just in parallel ahead of time? 169 00:15:34.140 --> 00:15:42.170 Shelley Doljack: Yes, so we… you run the… you migrate the users first, and then you migrate your roles, and it's spelled out in the… 170 00:15:42.310 --> 00:15:47.629 Shelley Doljack: that example wiki page, the wiki page of example will copy to Eureka Migration. 171 00:15:50.690 --> 00:15:59.699 Shelley Doljack: the, I should say that you have to recreate the credentials for users that log in with a username and password. 172 00:16:00.380 --> 00:16:04.749 Ingolf Kuss: Can you post the wiki page? I haven't occupied myself with the migration. 173 00:16:05.480 --> 00:16:06.610 Ingolf Kuss: At all. 174 00:16:06.780 --> 00:16:09.260 Shelley Doljack: Yeah, let me try to find it. 175 00:16:11.230 --> 00:16:17.219 Shelley Doljack: I mean, I just searched the wiki, for example, Okapi, something. 176 00:16:17.790 --> 00:16:25.049 Shelley Doljack: There, this… this is the wiki page. 177 00:16:26.910 --> 00:16:30.680 Shelley Doljack: So, it's, like, Phase 7. 178 00:16:33.840 --> 00:16:40.170 Shelley Doljack: Perform users migration, perform roles migration. 179 00:16:42.660 --> 00:16:43.660 Ingolf Kuss: Okay… 180 00:16:43.870 --> 00:16:51.189 Shelley Doljack: There's also some stuff in here, I'm not sure if it's linked up correctly or linked from the pages, but… 181 00:16:52.020 --> 00:16:58.299 Shelley Doljack: There's also this configuring RTR for Eureka. 182 00:16:58.300 --> 00:17:03.500 Ingolf Kuss: That's interesting, I only occupied myself with the point Kubernetes example deployment. 183 00:17:03.920 --> 00:17:04.520 Ingolf Kuss: But this isn't. 184 00:17:04.520 --> 00:17:05.570 Shelley Doljack: Oh, really? 185 00:17:05.579 --> 00:17:12.709 Ingolf Kuss: Yeah. But this is parallel to this. There's also a point migration to Eureka. What's the difference to the 186 00:17:13.589 --> 00:17:16.459 Ingolf Kuss: Example of a copy to Eureka migration? 187 00:17:19.359 --> 00:17:28.179 Shelley Doljack: I think the difference is that you have, this Phase 7 year with Users migration and roles migration. 188 00:17:28.180 --> 00:17:28.790 Ingolf Kuss: Okay. 189 00:17:33.560 --> 00:17:41.740 Shelley Doljack: Oh, and then another difference is that, you know, you're already starting with an OCopy-based deployment, so you have to… 190 00:17:42.580 --> 00:17:48.959 Shelley Doljack: Get rid of everything. And they do say, if you scroll up a bit, or scroll through this page. 191 00:17:49.160 --> 00:17:55.470 Shelley Doljack: They do say to make sure you're not running modLogin, modLogin SAML, mod auth token, and it'll copy modules. 192 00:17:58.610 --> 00:18:07.700 Shelley Doljack: So… Because they don't exist anymore. I mean, they're not used anymore in… In Eureka. 193 00:18:09.590 --> 00:18:15.830 Shelley Doljack: There's also an SSO configuration… page 194 00:18:23.690 --> 00:18:27.620 Shelley Doljack: There. I put in the chat, SSO configuration. 195 00:18:28.500 --> 00:18:35.650 Shelley Doljack: Yeah, so, I mean, we have our users, they log in with SSO, 196 00:18:35.800 --> 00:18:43.249 Shelley Doljack: But we do have users that have a username and password, so part of, our upgrade after we 197 00:18:43.860 --> 00:18:52.640 Shelley Doljack: you know, move the users over, we're just gonna recreate their passwords. You could recreate it in the settings developer… 198 00:18:52.990 --> 00:18:56.430 Shelley Doljack: Password. 199 00:18:56.590 --> 00:19:02.729 Shelley Doljack: You know what I'm talking about? In Stripes, that… or you could just go to KeyClick and create them there, too. 200 00:19:03.250 --> 00:19:09.680 Shelley Doljack: I think Key Cloak, at this point, is the, source of truth for… 201 00:19:10.080 --> 00:19:13.100 Shelley Doljack: User passwords and things like that. 202 00:19:13.890 --> 00:19:14.660 Ingolf Kuss: Okay. 203 00:19:14.850 --> 00:19:16.170 Shelley Doljack: user credentials. 204 00:19:19.440 --> 00:19:22.180 Shelley Doljack: Josh, do you have anything else to add? 205 00:19:23.410 --> 00:19:24.320 josh greben: Nope. 206 00:19:28.790 --> 00:19:32.480 Ingolf Kuss: Okay… Actually, we were still in the… 207 00:19:33.450 --> 00:19:39.359 Ingolf Kuss: introduction. So, Maraud, if you want to say a few words to your person, you don't have to, but I'll give you the… 208 00:19:39.360 --> 00:19:40.090 Mahrad Zoonematkermani: No, no, absolutely. 209 00:19:40.090 --> 00:19:40.650 Ingolf Kuss: What's your energy? 210 00:19:40.650 --> 00:19:42.480 Mahrad Zoonematkermani: Thank you. Yes, thanks. 211 00:19:42.660 --> 00:19:53.970 Mahrad Zoonematkermani: Yeah, Meredith, I, started working at LRZ together with Florian, Q4 of last year. 212 00:19:54.220 --> 00:19:56.860 Mahrad Zoonematkermani: So I'm pretty new to folio. 213 00:19:57.950 --> 00:20:05.500 Mahrad Zoonematkermani: But I've worked a lot with cloud technologies, and… Kubernetes in particular, so… 214 00:20:06.940 --> 00:20:16.320 Mahrad Zoonematkermani: That's why I jumped in and, helped with the Kubernetes stuff right away, and yeah, hopefully… 215 00:20:17.190 --> 00:20:21.139 Mahrad Zoonematkermani: will soon also be on the path to migration to Eureka. 216 00:20:21.990 --> 00:20:23.840 Mahrad Zoonematkermani: Don't want to jinx it, though. 217 00:20:23.840 --> 00:20:24.370 Ingolf Kuss: True. 218 00:20:26.000 --> 00:20:26.690 Mahrad Zoonematkermani: Yeah, thanks. 219 00:20:26.690 --> 00:20:27.450 Ingolf Kuss: Cute. 220 00:20:30.130 --> 00:20:37.240 Ingolf Kuss: Okay, I have made some… some progress with my installation. 221 00:20:38.110 --> 00:20:45.730 Ingolf Kuss: To recall, I was not able to do the entitlement. 222 00:20:46.530 --> 00:20:56.370 Ingolf Kuss: I'm still not able to, but I found one error… Namely… 223 00:20:58.020 --> 00:21:01.559 Ingolf Kuss: in the discovery file. It's called Discovery… 224 00:21:03.460 --> 00:21:08.619 Ingolf Kuss: You have to create it yourself. You have all the modules, the module… 225 00:21:08.800 --> 00:21:13.990 Ingolf Kuss: IDs, the module versions, and you have to create 226 00:21:15.290 --> 00:21:17.389 Ingolf Kuss: I think it's called the URL… 227 00:21:18.180 --> 00:21:21.689 Ingolf Kuss: And the location, or something I can show you, and . 228 00:21:21.690 --> 00:21:23.800 Mahrad Zoonematkermani: I have a script for that, if you want. 229 00:21:23.920 --> 00:21:30.519 Mahrad Zoonematkermani: You can use the application discovery, extract the information you need, and add the URL to it, or rather, locate. 230 00:21:30.520 --> 00:21:33.090 Ingolf Kuss: Yeah, you can share, I have my… 231 00:21:33.520 --> 00:21:33.960 Mahrad Zoonematkermani: Nope. 232 00:21:33.960 --> 00:21:35.580 Ingolf Kuss: Old script, but . 233 00:21:36.120 --> 00:21:42.119 Mahrad Zoonematkermani: Oh, no, if you have a script, I'll just show you later on what I did, and then you can adopt it. 234 00:21:42.120 --> 00:21:46.490 Ingolf Kuss: Let me, open a new… Good fish. 235 00:21:46.720 --> 00:21:52.480 Ingolf Kuss: Anyway, in the URL, you have to add the… The version number. 236 00:21:52.700 --> 00:21:54.839 Ingolf Kuss: And this, I didn't do. 237 00:21:57.510 --> 00:22:09.770 Ingolf Kuss: So… I have organized like this, and I want to entitle a platform minimal 2 or 30… So… 238 00:22:11.050 --> 00:22:17.649 Ingolf Kuss: And it was… Like this minus 301, this was missing. 239 00:22:18.000 --> 00:22:24.170 Ingolf Kuss: And then, therefore, I have gotten the messages, cannot connect. 240 00:22:24.660 --> 00:22:28.529 Ingolf Kuss: 2HT… this is a bad example, not… let's take a… 241 00:22:28.760 --> 00:22:30.890 Ingolf Kuss: The lock-in model, a regular model. 242 00:22:31.240 --> 00:22:34.969 Ingolf Kuss: cannot connect to HTTP mod settings because it's called 243 00:22:35.610 --> 00:22:39.150 Ingolf Kuss: HTTP mod setting 120, you know? 244 00:22:40.880 --> 00:22:41.949 Ingolf Kuss: You know? So… 245 00:22:41.950 --> 00:22:43.730 Mahrad Zoonematkermani: Oh, yeah, so it's a different problem. 246 00:22:43.730 --> 00:22:48.230 Ingolf Kuss: Zasha Dietrich told me this. You have to be able 2… 247 00:22:48.750 --> 00:22:54.960 Ingolf Kuss: to log into your port, and to do a thing or curl to this address. It has to exist. 248 00:22:55.490 --> 00:22:59.670 Ingolf Kuss: I can show you… If you don't mind. 249 00:23:01.500 --> 00:23:04.059 Ingolf Kuss: Find it. So you go to some port… 250 00:23:09.330 --> 00:23:14.080 Ingolf Kuss: Some regular module, mod users… to execute… 251 00:23:15.730 --> 00:23:17.929 Ingolf Kuss: I think this does not have curl. 252 00:23:18.630 --> 00:23:23.490 Ingolf Kuss: This is just a… Then you have to drive with VGAT. 253 00:23:27.130 --> 00:23:29.769 Ingolf Kuss: And it doesn't work, actually, I don't understand why. 254 00:23:30.150 --> 00:23:32.599 Mahrad Zoonematkermani: Shouldn't there be a port number at the end? 255 00:23:33.300 --> 00:23:35.850 Ingolf Kuss: Oh, yes, you're right. 256 00:23:37.360 --> 00:23:39.530 Ingolf Kuss: This one? Yeah, connecting. 257 00:23:39.840 --> 00:23:40.890 Shelley Doljack: But you also should 258 00:23:41.220 --> 00:23:48.220 Shelley Doljack: Remember that that address is actually the name of the service that you have running in your cluster. 259 00:23:49.830 --> 00:24:04.340 Shelley Doljack: Yes. I mention that because you have hyphens between the version number See, your ID is modsettings-1.2.0. 260 00:24:04.340 --> 00:24:05.339 Ingolf Kuss: Yeah, that's correct. 261 00:24:05.340 --> 00:24:09.300 Shelley Doljack: I think that… the name would be Mod Setting. 262 00:24:09.300 --> 00:24:10.029 Ingolf Kuss: I don't know what's… 263 00:24:10.030 --> 00:24:13.369 Shelley Doljack: 1.2.0 port 8082. 264 00:24:15.370 --> 00:24:18.240 Mahrad Zoonematkermani: I mean, I'm also using the same version. 265 00:24:18.240 --> 00:24:19.680 Ingolf Kuss: To your means and name? 266 00:24:20.090 --> 00:24:23.479 Mahrad Zoonematkermani: Can you take a look at your services and see… 267 00:24:24.660 --> 00:24:26.890 Mahrad Zoonematkermani: If you have such a service. Maybe the service. 268 00:24:26.890 --> 00:24:29.850 Ingolf Kuss: No, I don't understand this anymore. Yes, we will do this. 269 00:24:29.850 --> 00:24:30.869 Mahrad Zoonematkermani: Yeah, on the left. 270 00:24:30.870 --> 00:24:33.339 Ingolf Kuss: Because it's… because it had worked. 271 00:24:34.850 --> 00:24:42.000 Ingolf Kuss: So if I look for mod settings in my namespace… There you go, every service… 272 00:24:42.360 --> 00:24:43.640 Mahrad Zoonematkermani: It's go on one city. 273 00:24:44.300 --> 00:24:49.970 Ingolf Kuss: Yeah, yeah, it should be there, so it should be… Oh, that's like… 274 00:24:50.150 --> 00:24:57.439 Shelley Doljack: Okay, so I guess we're just wanting to confirm that the name of the service is what you write out there in your. 275 00:24:57.440 --> 00:24:58.420 Ingolf Kuss: Oh, yeah. 276 00:24:58.420 --> 00:24:59.340 Shelley Doljack: your script. 277 00:25:02.530 --> 00:25:04.640 Ingolf Kuss: So I think this works, but there's a… 278 00:25:05.780 --> 00:25:08.949 Ingolf Kuss: What's the address? Oh, I have to go back to some port. 279 00:25:13.050 --> 00:25:18.520 Ingolf Kuss: Anyway, if… We don't have to… Get to in more detail. 280 00:25:18.760 --> 00:25:24.259 Ingolf Kuss: Because it's… Why does all this not work? I don't know what's going on. 281 00:25:24.260 --> 00:25:25.350 Mahrad Zoonematkermani: port number. 282 00:25:28.500 --> 00:25:35.100 Ingolf Kuss: Different port number… 8082? I don't know, this has worked, I don't know. 283 00:25:36.000 --> 00:25:42.850 Mahrad Zoonematkermani: Yeah, but it's registering internal server error, so that means the server is there, it's just not… 284 00:25:43.590 --> 00:25:44.230 Ingolf Kuss: Oh, your means is. 285 00:25:44.230 --> 00:25:44.700 Mahrad Zoonematkermani: Even worse. 286 00:25:44.700 --> 00:25:45.180 Ingolf Kuss: joked. 287 00:25:45.180 --> 00:25:48.920 Mahrad Zoonematkermani: Yes, if you look at the mod settings pod. 288 00:25:48.920 --> 00:25:49.350 Ingolf Kuss: Oh. 289 00:25:49.350 --> 00:25:52.430 Mahrad Zoonematkermani: Logs, you should see why it registered the error. 290 00:25:54.440 --> 00:25:55.820 Ingolf Kuss: What can I scroll? 291 00:25:57.630 --> 00:25:58.480 Ingolf Kuss: I don't know. 292 00:26:00.220 --> 00:26:03.409 Ingolf Kuss: This doesn't show me the whole screen, you know? 293 00:26:04.410 --> 00:26:05.990 Ingolf Kuss: But I don't know how to… 294 00:26:08.890 --> 00:26:14.389 Ingolf Kuss: Let's go to my current error. Just for a side remark, I installed 295 00:26:14.750 --> 00:26:19.030 Ingolf Kuss: another Kubernetes cluster for some other project, not folio, and 296 00:26:19.460 --> 00:26:27.809 Ingolf Kuss: I wanted to install the Kubernetes dashboard, and believe it or not, it doesn't exist anymore. You cannot install this anymore. 297 00:26:27.970 --> 00:26:32.440 Ingolf Kuss: The new product is called Headlamp, which is quite nice. 298 00:26:33.960 --> 00:26:34.899 Ingolf Kuss: I've played with. 299 00:26:34.900 --> 00:26:36.090 Shelley Doljack: headlamp. 300 00:26:37.200 --> 00:26:37.750 Shelley Doljack: Yeah. 301 00:26:37.750 --> 00:26:38.380 Ingolf Kuss: You would? 302 00:26:38.710 --> 00:26:41.930 Shelley Doljack: I said I've played with headlamps once upon a time. 303 00:26:43.260 --> 00:26:43.940 Ingolf Kuss: Okay, cool. 304 00:26:43.940 --> 00:26:47.600 Florian Kreft (LRZ): We also have some test installations for that, 305 00:26:48.240 --> 00:27:01.949 Florian Kreft (LRZ): I think in the most cases, it has various similar features, some buttons moved around quite a bit, you have to… some search functions work a bit different, but in the end, we probably will have to go through that, I think. 306 00:27:03.600 --> 00:27:16.520 Ingolf Kuss: Yeah, or I thought I installed Rancher UI, because many of you have Rancher, including the GBV guys, but I started this, and I found it too complicated, because Rancher is… 307 00:27:16.730 --> 00:27:23.150 Ingolf Kuss: doing everything via HTTPS. You have to create certificates and set this all up. 308 00:27:23.530 --> 00:27:28.470 Ingolf Kuss: Correctly, I thought it's too complicated, so… Put it aside. 309 00:27:28.470 --> 00:27:34.289 Mahrad Zoonematkermani: Rancher only works great if you actually, bootstrapped your cluster with RKE. 310 00:27:34.510 --> 00:27:35.950 Ingolf Kuss: Okay. So… 311 00:27:36.370 --> 00:27:38.970 Mahrad Zoonematkermani: I mean, I might be wrong. Please don't quote me on. 312 00:27:38.970 --> 00:27:46.259 Ingolf Kuss: Yeah, no, no, I believe. There's a… I mean, you can connect to an existing K3S, in my case, cluster. 313 00:27:46.390 --> 00:27:49.350 Ingolf Kuss: According to the documentation. 314 00:27:49.350 --> 00:27:55.270 Mahrad Zoonematkermani: Yeah, but K3S is also kind of, like, from the rancher as… 315 00:27:55.270 --> 00:27:56.120 Ingolf Kuss: Oh, really? 316 00:27:56.120 --> 00:27:56.730 Mahrad Zoonematkermani: Yep, yep. 317 00:27:56.730 --> 00:28:03.759 Ingolf Kuss: Oh, yes, right, because, yeah, I have passed this ETC wrench, yeah, okay. Thank you, I was wondering, yeah. 318 00:28:04.730 --> 00:28:11.549 Ingolf Kuss: Yeah, so it should work, but again, the third time, I found this too complicated. So, at the moment. 319 00:28:12.600 --> 00:28:14.620 Ingolf Kuss: I have the following error. 320 00:28:15.870 --> 00:28:22.820 Ingolf Kuss: Maybe I'll just show you… Failed to perform post-tenant call. 321 00:28:22.960 --> 00:28:24.689 Ingolf Kuss: Could not create admin. 322 00:28:24.890 --> 00:28:29.250 Ingolf Kuss: What does this mean? Illegal state exception. What kind of admin? 323 00:28:29.760 --> 00:28:32.110 Shelley Doljack: Well, it says Value 400. 324 00:28:33.820 --> 00:28:34.420 Shelley Doljack: Do post. 325 00:28:34.420 --> 00:28:37.059 Ingolf Kuss: Oh, what is this? 400? It's just a… 326 00:28:37.210 --> 00:28:40.069 Shelley Doljack: Maybe it's, the request was bad. 327 00:28:40.680 --> 00:28:42.740 Ingolf Kuss: Bad request, yeah, bad request. 328 00:28:44.070 --> 00:28:48.049 Ingolf Kuss: So… I wanted to ask another question, because, 329 00:28:48.480 --> 00:28:52.400 Ingolf Kuss: I find doing the entitlement, it scales up 330 00:28:52.880 --> 00:28:57.260 Ingolf Kuss: 3 of my containers. I have the pod auto-scaling on. 331 00:28:57.770 --> 00:29:03.340 Ingolf Kuss: Namely, those ones, which is not so surprising, because they are heavily involved. 332 00:29:03.910 --> 00:29:06.750 Ingolf Kuss: And what uses KeyCloud? And it says… 333 00:29:07.410 --> 00:29:07.830 Shelley Doljack: So. 334 00:29:07.830 --> 00:29:13.909 Ingolf Kuss: reason, CPU, resource utilization above target. So what I wanted to ask you is, how have you set your. 335 00:29:14.660 --> 00:29:27.279 Shelley Doljack: We don't… okay. I would highly recommend turning that off while you're entitling. We are only deploying one of everything. 336 00:29:27.430 --> 00:29:42.420 Shelley Doljack: for this process, for the entitlement. And then after, we're gonna scale up our stuff to what it should be. Like, 2. We're gonna do two. But we don't… we don't have horizontal pod… 337 00:29:42.580 --> 00:29:49.589 Shelley Doljack: Autoscaling, yeah, we've never… we've never configured at that. 338 00:29:49.590 --> 00:29:58.920 Ingolf Kuss: But why should that be bad, Shelly? I mean, it says that it has not enough memory, and it does a second container and a third container, and it still does not work, so my con… 339 00:29:59.130 --> 00:30:00.090 Ingolf Kuss: Suspicion this? 340 00:30:00.090 --> 00:30:00.680 Shelley Doljack: Jeep? 341 00:30:00.870 --> 00:30:01.900 Ingolf Kuss: This is not a… 342 00:30:02.120 --> 00:30:06.729 Ingolf Kuss: enough memory, so what… what… can I ask you, what are your limits here? This is. 343 00:30:07.850 --> 00:30:08.869 Florian Kreft (LRZ): In such cases… 344 00:30:08.870 --> 00:30:10.600 Ingolf Kuss: It was a ski float? 345 00:30:10.600 --> 00:30:17.729 Shelley Doljack: In our, our repo, and I also have a spreadsheet. Let me… English… 346 00:30:17.730 --> 00:30:30.800 Florian Kreft (LRZ): Just like a general point on that. If you have multiple containers of the same applications which the workload gets distributed around on, this just complicates all kinds of problems. 347 00:30:31.300 --> 00:30:33.890 Florian Kreft (LRZ): It's just, like, maybe one request goes to one. 348 00:30:33.890 --> 00:30:40.320 Ingolf Kuss: I don't understand. I thought the whole point to do this on the Kubernetes cluster is that you have the horizontal scaling. 349 00:30:40.320 --> 00:31:00.710 Florian Kreft (LRZ): If everything works right, yes, but if you're still in the troubleshooting stage and have not done that with one container, it's a bad idea to first scale up and then troubleshoot. It's just not… it's not the right way. The better way to, like, for first testing is simply increase the request for a single pod, so that that doesn't crash. 350 00:31:00.710 --> 00:31:05.000 Florian Kreft (LRZ): So if you have, like, one pod which runs into out-of-memory issues. 351 00:31:05.000 --> 00:31:05.780 Ingolf Kuss: And so… 352 00:31:05.780 --> 00:31:14.899 Florian Kreft (LRZ): stuff like entitlement, they just need more resources, so each one of them needs more resources to do their task. If in the end, then you have… 353 00:31:14.900 --> 00:31:32.490 Florian Kreft (LRZ): like, a giant instance with lots of users in parallel using this, then the horizontal autoscaling might start to make sense, but not in the initial troubleshooting phase. It's just, like, it makes things just more… it's just another error source. 354 00:31:32.710 --> 00:31:34.100 Florian Kreft (LRZ): Just… yeah, yeah. 355 00:31:34.100 --> 00:31:42.849 Florian Gleixner: And to complete this, if you enable horizontal port autoscaling, you have to be really sure that you're 356 00:31:43.230 --> 00:31:47.739 Florian Gleixner: Your pod is up and running before the service connects to the pod, so… 357 00:31:48.090 --> 00:31:55.129 Florian Gleixner: If your readiness probes are not configured correctly, then autoscaling may break. 358 00:31:55.280 --> 00:32:02.040 Florian Gleixner: Everything, because, it starts up a pod, and the pod is not ready, but requests go to the not-ready pod. 359 00:32:02.920 --> 00:32:03.580 Florian Kreft (LRZ): Yep. 360 00:32:04.070 --> 00:32:05.799 Florian Gleixner: That's also a problem. 361 00:32:06.620 --> 00:32:12.280 Florian Gleixner: Before using this, you… you have to… test everything. 362 00:32:12.280 --> 00:32:15.009 Mahrad Zoonematkermani: Sorry to interrupt, but I think he got disconnected. 363 00:32:15.330 --> 00:32:18.699 Florian Kreft (LRZ): No. Yeah, that's… you should be here to answer. 364 00:32:19.100 --> 00:32:22.909 Florian Kreft (LRZ): No points, yeah. Otherwise, they don't make sense. 365 00:32:24.240 --> 00:32:30.790 Florian Kreft (LRZ): But I think two people got disconnected, right? Was the Josh? Oh, no, the Josh, Josh, Josh. No, Josh had to be… 366 00:32:31.190 --> 00:32:31.830 Florian Gleixner: Hmm. 367 00:32:38.390 --> 00:32:49.269 Shelley Doljack: Okay, I, I'm putting a spreadsheet that I… Let me share my screen. 368 00:32:50.140 --> 00:32:56.220 Shelley Doljack: Share button… Okay, 369 00:32:56.790 --> 00:33:04.050 Shelley Doljack: I don't know if it's too small. I put this in the… the SysOps management SIG drive. 370 00:33:06.800 --> 00:33:10.559 Shelley Doljack: This is for… I… I don't… let me give you the link. 371 00:33:10.660 --> 00:33:14.820 Shelley Doljack: I don't know if… How many people have access to this? 372 00:33:16.570 --> 00:33:19.450 Shelley Doljack: Where's my chat? 373 00:33:22.580 --> 00:33:30.149 Shelley Doljack: Okay, so I got from, from this wiki page. 374 00:33:31.720 --> 00:33:35.119 Shelley Doljack: This was pointed out to me from… 375 00:33:35.760 --> 00:33:53.169 Shelley Doljack: somebody in the performance testing group, because I was asking about, like, what the heck? Like, what do you… because we needed to ask for more RAM, in our cluster in order to deploy Eureka. And so they… I copied this, 376 00:33:53.920 --> 00:34:03.400 Shelley Doljack: this thing here that has all the modules and, you know, their resourcing stuff, and I put it in a spreadsheet, and then I, like. 377 00:34:03.540 --> 00:34:09.389 Shelley Doljack: tailored it to, sorry, I just copied this, and so it's kind of, like, off. 378 00:34:10.159 --> 00:34:18.899 Shelley Doljack: Let me do a… freeze the top row. So this is what… hi, sorry, I'm showing this. 379 00:34:18.900 --> 00:34:19.300 Ingolf Kuss: That's huge. 380 00:34:19.300 --> 00:34:23.979 Shelley Doljack: You're asking about, what do you have for your memory and stuff, and so… 381 00:34:23.989 --> 00:34:24.919 Ingolf Kuss: Oh, thank you. 382 00:34:24.920 --> 00:34:30.960 Shelley Doljack: There's, at the bottom of this, I put this in the SysOps SIG drive. 383 00:34:30.969 --> 00:34:31.649 Ingolf Kuss: Cool. 384 00:34:31.900 --> 00:34:45.529 Shelley Doljack: And I got it from this wiki page. And this… and I needed… I wanted to calculate, because we had to increase the amount of memory in… in order to, deploy Eureka and have a good headroom for stuff. 385 00:34:45.659 --> 00:34:59.580 Shelley Doljack: And so, at the bottom of the wiki page, there's a whole bunch of stuff listed. I tailored it down to what we are actually deploying, so we don't have all the edge modules. 386 00:34:59.680 --> 00:35:08.419 Shelley Doljack: And I kind of ignored some of these columns that I brought over, and just concentrated on, this column. 387 00:35:08.700 --> 00:35:17.790 Shelley Doljack: In this column… And I guess, you know, all the sidecars, they're gonna have the same… Bye. 388 00:35:19.200 --> 00:35:24.720 Shelley Doljack: I don't know, I didn't, they're… they all… we all have them all set to the same thing. 389 00:35:24.860 --> 00:35:31.390 Shelley Doljack: I don't know why it says Rene here. So this is made to be, like, we're running two of these. 390 00:35:31.710 --> 00:35:36.890 Shelley Doljack: we're… Actually, we're… we're only running one Kong, 391 00:35:39.470 --> 00:35:41.870 Shelley Doljack: Yeah, we're only gonna run one Kong. 392 00:35:42.110 --> 00:36:01.059 Shelley Doljack: So, I'm gonna update that. So this is just… this column just multiplies this one by that one. So these are the memory hard limits, so the, I guess, the resources limit, and the resources request that we're doing for each. And, 393 00:36:02.300 --> 00:36:04.180 Shelley Doljack: Why did I have this highlighted? 394 00:36:04.340 --> 00:36:05.130 Shelley Doljack: I think because… 395 00:36:05.130 --> 00:36:06.299 Ingolf Kuss: Because I asked for this. 396 00:36:06.300 --> 00:36:11.849 Shelley Doljack: They didn't have anything… they didn't have mod logging geek logos, like, why is that not there? 397 00:36:11.960 --> 00:36:26.840 Shelley Doljack: Anyways, end of story is, like, we asked for 290 megaby… gigabytes of RAM for our namespace in order to deploy Eureka. And then, like, this is… this is kind of the stuff that… 398 00:36:26.840 --> 00:36:27.420 Ingolf Kuss: Oh, really? 399 00:36:27.420 --> 00:36:36.329 Shelley Doljack: infrastructure stuff that we have running in the same namespace. Yeah, so I'll put a copy of this, 400 00:36:36.660 --> 00:36:38.240 Shelley Doljack: the link. 401 00:36:38.240 --> 00:36:39.000 Ingolf Kuss: Aww. 402 00:36:39.000 --> 00:36:39.360 Shelley Doljack: That. 403 00:36:39.360 --> 00:36:41.960 Ingolf Kuss: So, I think you showed one to… 404 00:36:41.960 --> 00:36:43.529 Shelley Doljack: You should look at that. 405 00:36:44.400 --> 00:36:50.050 Ingolf Kuss: I actually have 1,042, 24, and 512. 406 00:36:51.100 --> 00:36:59.070 Ingolf Kuss: But this is memory, what did you put for CPU? I think the problem was CPU. I have 100M, just for the… 407 00:36:59.330 --> 00:37:00.610 Mahrad Zoonematkermani: That would be… 408 00:37:00.610 --> 00:37:05.689 Ingolf Kuss: This means 100 milli, isn't it? So, 0.1, correct? 409 00:37:05.950 --> 00:37:10.210 Ingolf Kuss: Well, if it says CPU, 100M. 410 00:37:10.370 --> 00:37:12.899 Mahrad Zoonematkermani: Yes, that's correct. 411 00:37:13.280 --> 00:37:14.160 Mahrad Zoonematkermani: Too low. 412 00:37:14.350 --> 00:37:18.210 Ingolf Kuss: Yeah, but this is the limit which I have, and 413 00:37:18.800 --> 00:37:21.510 Ingolf Kuss: Shelly, did you show this? Do you have it in your sheet? 414 00:37:21.510 --> 00:37:27.680 Shelley Doljack: I don't… I… no, I didn't… I didn't mess with the CPU column, because we already got that, 415 00:37:28.550 --> 00:37:30.820 Shelley Doljack: squared away, but we do… this is… 416 00:37:31.280 --> 00:37:36.810 Ingolf Kuss: But what do you have there? Then you have 150, like, the default, or what? 417 00:37:37.940 --> 00:37:49.220 Shelley Doljack: So this is what we use as a default for all of… all of our modules. Okay. And some of… this is… so this… this value… sorry if it's too small. This is what we do. 418 00:37:49.540 --> 00:37:51.740 Shelley Doljack: all of our modules, like, that's the base. 419 00:37:51.740 --> 00:37:52.450 Ingolf Kuss: Oh… 420 00:37:52.450 --> 00:37:53.320 Shelley Doljack: And then we have… 421 00:37:53.320 --> 00:37:53.860 Ingolf Kuss: Okay. 422 00:37:53.860 --> 00:37:54.330 Shelley Doljack: Tom? 423 00:37:54.330 --> 00:37:54.870 Ingolf Kuss: Thank you. 424 00:37:54.870 --> 00:37:59.599 Shelley Doljack: where, we might have… different. 425 00:38:00.280 --> 00:38:05.610 Shelley Doljack: So, like, for mod bulk operations, we are giving it a lot more. 426 00:38:06.160 --> 00:38:08.520 Shelley Doljack: Wait, this doesn't look. 427 00:38:08.520 --> 00:38:09.270 Ingolf Kuss: Whoa. 428 00:38:09.590 --> 00:38:11.099 Shelley Doljack: That doesn't look right at all. 429 00:38:11.530 --> 00:38:14.129 Ingolf Kuss: Limits is higher, isn't it? So you had to… 430 00:38:15.380 --> 00:38:21.339 Ingolf Kuss: Requests 150 and limits 200, what was the default? Can you show it again? 431 00:38:21.340 --> 00:38:24.209 Shelley Doljack: The default we put under common here… 432 00:38:25.080 --> 00:38:26.570 Ingolf Kuss: Limits to 100. 433 00:38:27.440 --> 00:38:30.339 Ingolf Kuss: Okay, cool, I have to double that. 434 00:38:31.340 --> 00:38:35.439 Shelley Doljack: And some of them we've lowered, and some of them we've increased a lot. 435 00:38:36.550 --> 00:38:37.320 Shelley Doljack: So… 436 00:38:39.110 --> 00:38:42.079 Ingolf Kuss: And for the sidecars, they have their own. 437 00:38:42.440 --> 00:38:42.930 Shelley Doljack: Oh, yeah. 438 00:38:42.930 --> 00:38:45.980 Ingolf Kuss: Limits, what resources? What did you put? 439 00:38:48.190 --> 00:38:51.070 Shelley Doljack: I think it's… the same? 440 00:38:52.340 --> 00:38:54.109 Shelley Doljack: I don't know, for the psych curse. 441 00:38:55.240 --> 00:38:56.940 Shelley Doljack: Let me share again. 442 00:38:57.180 --> 00:38:59.979 Ingolf Kuss: Addresses of 25 and 75. 443 00:39:00.840 --> 00:39:09.060 Ingolf Kuss: CPU… Okay, sidecar, resources, limits… 150? 444 00:39:10.480 --> 00:39:15.090 Ingolf Kuss: Yeah, that's a… yeah, that's a double value than I have, okay? 445 00:39:15.500 --> 00:39:16.920 Ingolf Kuss: And 100. 446 00:39:18.180 --> 00:39:34.469 Shelley Doljack: And, like, if you're doing horizontal pod auto-scaling, what, you know, when these modules come up, they use a lot, like, when they first start up, so it's… you're really, like, throwing in more comp… 447 00:39:34.470 --> 00:39:40.310 Shelley Doljack: complication than you need to, I think. That's… 448 00:39:40.630 --> 00:39:41.150 Ingolf Kuss: No… 449 00:39:43.080 --> 00:39:46.980 Ingolf Kuss: I've done this for a lot of modules, I have to redeploy, or I don't know if I want to. 450 00:39:47.090 --> 00:39:48.410 Ingolf Kuss: Go through this. 451 00:39:48.940 --> 00:39:52.169 Florian Gleixner: Why do you… why do you set your CPU limits so low? 452 00:39:53.250 --> 00:39:56.020 Florian Gleixner: Do you have problems with CPU, or… 453 00:39:56.800 --> 00:39:57.630 Florian Gleixner: Do you have. 454 00:39:57.630 --> 00:39:58.310 Ingolf Kuss: Cool. 455 00:39:58.770 --> 00:39:59.490 Florian Gleixner: Yeah, you. 456 00:40:00.850 --> 00:40:03.030 Florian Gleixner: Or both? Because. 457 00:40:03.030 --> 00:40:03.570 Shelley Doljack: Question? 458 00:40:03.570 --> 00:40:05.120 Ingolf Kuss: I… 459 00:40:05.120 --> 00:40:06.499 Florian Kreft (LRZ): The CPU was a little… 460 00:40:06.500 --> 00:40:11.219 Ingolf Kuss: Well, either it's a default, or I… Never cared about this, too. 461 00:40:11.220 --> 00:40:18.700 Florian Gleixner: Yeah, maybe… maybe for CPU limits, the problem is if you, if you limit them, then the 462 00:40:18.980 --> 00:40:21.099 Florian Gleixner: modules get slow, so… 463 00:40:21.100 --> 00:40:21.650 Florian Kreft (LRZ): Cup. 464 00:40:23.050 --> 00:40:23.410 Florian Gleixner: But the… 465 00:40:23.410 --> 00:40:25.959 Ingolf Kuss: Yeah, but you have to have another, 466 00:40:26.130 --> 00:40:34.529 Ingolf Kuss: resources, I don't have enough resources on my… maybe we can talk about this, if I have enough resources for this, maybe we can… 467 00:40:34.890 --> 00:40:40.539 Florian Gleixner: Have you… have you… can you… can you look at the… to see you. Sure. 468 00:40:41.060 --> 00:40:43.359 Ingolf Kuss: No, I mean for this, 469 00:40:44.330 --> 00:40:54.450 Ingolf Kuss: I think it's under notes in this, here, I mean, I have a… for CPU… 470 00:40:55.170 --> 00:40:57.750 Ingolf Kuss: Oh, it's crashed here, sorry. 471 00:40:58.670 --> 00:41:07.620 Ingolf Kuss: I have 3 nodes, and they have 50… 40GB RAM each, so that is 120, and I have 4 CPU. 472 00:41:07.820 --> 00:41:10.370 Ingolf Kuss: Each, so this is 12, and 473 00:41:10.520 --> 00:41:18.639 Ingolf Kuss: Since Shelley already said she has 200-something, so I can never match that, so… and at the moment. 474 00:41:18.850 --> 00:41:19.460 Florian Gleixner: Okay. 475 00:41:19.460 --> 00:41:24.979 Ingolf Kuss: It's like this… This is what I've said, it is 4 CPU per call. 476 00:41:26.150 --> 00:41:31.609 Ingolf Kuss: And that his limit is, yeah, it's around 4, he's 6 to 6'1". 477 00:41:32.570 --> 00:41:45.990 Ingolf Kuss: But this is reserve limits. I think these limits are never reached, so it should be fine. I still think it should be fine. These limits are never reached. This is what I have reserved. Don't know what's the English… 478 00:41:46.300 --> 00:41:48.960 Ingolf Kuss: Is this into… Yeah. 479 00:41:49.090 --> 00:41:51.500 Ingolf Kuss: But is it called Resoft in Europe? 480 00:41:54.170 --> 00:42:01.360 Mahrad Zoonematkermani: May I make a recommendation? Like, for… scheduling, and… 481 00:42:01.840 --> 00:42:12.359 Mahrad Zoonematkermani: For running your application in general, it's a very good idea to have requests And limits for your memory. 482 00:42:12.760 --> 00:42:18.660 Mahrad Zoonematkermani: But it's not a must to have a limit for your… . 483 00:42:19.570 --> 00:42:20.390 Florian Gleixner: CPU, yeah. 484 00:42:20.390 --> 00:42:25.449 Mahrad Zoonematkermani: CPU. Granted, you're not using HPA. 485 00:42:25.630 --> 00:42:31.930 Mahrad Zoonematkermani: Which you are. So, I would go with Shelley's. 486 00:42:31.930 --> 00:42:32.270 Florian Kreft (LRZ): Nope. 487 00:42:32.270 --> 00:42:39.209 Mahrad Zoonematkermani: recommendation to just not use HPA in general, because scaling out 488 00:42:39.610 --> 00:42:43.260 Mahrad Zoonematkermani: Especially for applications that run on Java. 489 00:42:43.710 --> 00:42:50.540 Mahrad Zoonematkermani: I wouldn't really recommend it, because Java uses a lot of baseline resources, so… 490 00:42:50.540 --> 00:42:52.690 Ingolf Kuss: I mean, HPA, high availability, or what is that? 491 00:42:52.690 --> 00:43:04.149 Mahrad Zoonematkermani: Horizontal pod autoscaler. We have horizontal pod autoscaler, which scales out, meaning that it adds more replicas of your pod. 492 00:43:05.070 --> 00:43:11.440 Mahrad Zoonematkermani: And we have vertical pod autoscalers, which is actually a good recommendation for your case. 493 00:43:11.550 --> 00:43:12.510 Mahrad Zoonematkermani: Bitch. 494 00:43:13.620 --> 00:43:24.340 Mahrad Zoonematkermani: increases the resources of the pod that you're currently running. So instead of giving you more pods, it changes your pod so that it's bigger. 495 00:43:24.570 --> 00:43:32.189 Mahrad Zoonematkermani: However, be careful if you want to use VPA, because for vertical pod autoscaler, if you're using an older version. 496 00:43:32.280 --> 00:43:47.460 Mahrad Zoonematkermani: version of Kubernetes, it constantly kills your pod, which is not something you want. But if you're on a newer version of Kubernetes, you don't have to worry about it, because this change happens on the fly without evicting your pod. 497 00:43:48.010 --> 00:43:56.419 Mahrad Zoonematkermani: Now… With that being said, the simplest solution for you, I would say, would be to go with, 498 00:43:57.090 --> 00:44:01.120 Mahrad Zoonematkermani: share this suggestion and not use HPA at all. 499 00:44:01.830 --> 00:44:07.689 Mahrad Zoonematkermani: or at least for your testing purposes, because as Florian, 500 00:44:09.330 --> 00:44:19.000 Mahrad Zoonematkermani: suggest that it adds more complications than being helpful. Usually, distributed systems are annoying. 501 00:44:19.930 --> 00:44:21.850 Mahrad Zoonematkermani: Just put it simply. Yeah. 502 00:44:22.260 --> 00:44:29.249 Mahrad Zoonematkermani: So, just get rid of that, and then don't even give it a CPU limit at all. 503 00:44:29.420 --> 00:44:31.219 Mahrad Zoonematkermani: That way, the bursts. 504 00:44:31.220 --> 00:44:31.660 Ingolf Kuss: Beautiful. 505 00:44:31.660 --> 00:44:33.439 Mahrad Zoonematkermani: During the startup. 506 00:44:33.540 --> 00:44:35.680 Ingolf Kuss: Would I have to give it a limit here. 507 00:44:35.850 --> 00:44:38.880 Mahrad Zoonematkermani: No, no, you don't have to. You can just remove the limit. 508 00:44:39.720 --> 00:44:43.490 Ingolf Kuss: This is my values file, this is members. 509 00:44:43.490 --> 00:44:45.050 Mahrad Zoonematkermani: CPU there is on. 510 00:44:45.050 --> 00:44:47.669 Ingolf Kuss: So what do you put here? Nothing like this, or what? 511 00:44:47.670 --> 00:44:48.290 Florian Kreft (LRZ): fruit, yeah. 512 00:44:48.290 --> 00:44:48.840 Mahrad Zoonematkermani: Ohio. 513 00:44:49.100 --> 00:44:50.000 Mahrad Zoonematkermani: That's it. 514 00:44:51.690 --> 00:44:53.650 Ingolf Kuss: Okay, interesting. 515 00:44:53.650 --> 00:44:54.940 Mahrad Zoonematkermani: Yep, so… 516 00:44:54.950 --> 00:44:55.570 Ingolf Kuss: That's… 517 00:44:55.570 --> 00:45:00.280 Mahrad Zoonematkermani: That would mean, when this comes up, okay, but how about the… 518 00:45:00.550 --> 00:45:05.939 Ingolf Kuss: I mean, if I scale this up, then I don't have enough CPU here, you know? 519 00:45:07.150 --> 00:45:10.959 Ingolf Kuss: telling me I have to comment it all out. Yeah, okay. 520 00:45:11.710 --> 00:45:17.599 Mahrad Zoonematkermani: No, no, but you don't have to worry about it, because your pods will compete for it on their own. 521 00:45:17.930 --> 00:45:19.669 Mahrad Zoonematkermani: So, not a big problem. 522 00:45:20.200 --> 00:45:20.540 Florian Kreft (LRZ): Yeah. 523 00:45:20.540 --> 00:45:21.219 Florian Gleixner: If you are. 524 00:45:21.220 --> 00:45:31.559 Mahrad Zoonematkermani: What you need to worry about is the scheduling part, because if you don't have enough CPU in your requests. 525 00:45:32.360 --> 00:45:47.429 Mahrad Zoonematkermani: when you sum up all of your requests and add them up, and when your nodes don't have that, then you cannot schedule your pods. But as soon as your pod is scheduled, it's not like RAM that it's going to kill your process, it's just gonna slow it down a bit. 526 00:45:47.550 --> 00:45:50.660 Mahrad Zoonematkermani: But by not setting a limit. 527 00:45:51.590 --> 00:46:05.290 Mahrad Zoonematkermani: Basically, the bursty jobs that would try to take a lot of resources from your machines would do their job if nothing else is doing. 528 00:46:05.290 --> 00:46:06.720 Ingolf Kuss: the limit, so you're saying I should do. 529 00:46:06.720 --> 00:46:08.349 Mahrad Zoonematkermani: No, no, request has to be done. 530 00:46:08.350 --> 00:46:09.350 Florian Kreft (LRZ): Just, yeah, recursive. 531 00:46:09.350 --> 00:46:10.830 Mahrad Zoonematkermani: The limit needs to be moved. 532 00:46:10.830 --> 00:46:12.699 Florian Kreft (LRZ): So the request is just… what? 533 00:46:12.700 --> 00:46:15.060 Ingolf Kuss: And what happens to these figures in here? 534 00:46:15.250 --> 00:46:16.760 Ingolf Kuss: facet request. 535 00:46:17.050 --> 00:46:21.560 Florian Kreft (LRZ): The request column remains the same, and the limit actually goes down. 536 00:46:21.560 --> 00:46:22.999 Ingolf Kuss: This is a request or what? 537 00:46:23.000 --> 00:46:24.500 Florian Kreft (LRZ): Yeah, anymore. Yeah, yeah. 538 00:46:25.810 --> 00:46:34.749 Ingolf Kuss: Yeah, but this is what I'm saying. If Shelly tells me I have to double this, this was a 50 before, then it goes below, higher than 4 here. 539 00:46:35.440 --> 00:46:42.070 Mahrad Zoonematkermani: Yes, but when you don't specify anything, Things will figure themselves out. 540 00:46:42.750 --> 00:46:43.320 Florian Kreft (LRZ): Yep. 541 00:46:44.060 --> 00:46:47.469 Ingolf Kuss: Don't specify… a specifier… 542 00:46:47.680 --> 00:46:50.200 Mahrad Zoonematkermani: And you don't specify any limits. 543 00:46:50.200 --> 00:46:51.540 Shelley Doljack: Kubernetty's magic. 544 00:46:51.540 --> 00:46:52.680 Florian Gleixner: Yeah. 545 00:46:52.680 --> 00:46:54.379 Ingolf Kuss: Don't specify limits. 546 00:46:54.380 --> 00:46:55.410 Florian Gleixner: So the request… 547 00:46:55.410 --> 00:46:58.039 Ingolf Kuss: What do you want me to do? And the memory doesn't matter, or what? 548 00:46:58.040 --> 00:47:00.489 Mahrad Zoonematkermani: No, no, memory has to be specified. 549 00:47:00.490 --> 00:47:03.309 Ingolf Kuss: So just say the limits, only the limits. 550 00:47:03.450 --> 00:47:04.439 Ingolf Kuss: The limit is… 551 00:47:04.440 --> 00:47:06.030 Mahrad Zoonematkermani: CPU removed, please. 552 00:47:06.610 --> 00:47:13.370 Mahrad Zoonematkermani: And that is if it doesn't have a default value. So you're using these in… 553 00:47:13.770 --> 00:47:14.510 Florian Kreft (LRZ): MV2. 554 00:47:14.510 --> 00:47:31.429 Mahrad Zoonematkermani: the Helm v2, and you're applying these locally, so you're not pointing to the repository, the Helm repository, but rather your install points to the local directory. If that's the case, then if you remove the limit, you're fine. 555 00:47:32.150 --> 00:47:33.130 Ingolf Kuss: Interesting. 556 00:47:34.800 --> 00:47:35.720 Mahrad Zoonematkermani: I mean, I hope. 557 00:47:35.720 --> 00:47:36.220 Ingolf Kuss: Oh, okay. 558 00:47:37.100 --> 00:47:42.620 Florian Kreft (LRZ): Because now, I mean, you should keep an eye on the actual nodes, so on the actual CPU load. 559 00:47:43.110 --> 00:47:44.350 Florian Kreft (LRZ): Also, if they're under… 560 00:47:44.470 --> 00:47:52.589 Ingolf Kuss: I keep… this is a good question, I don't know how to keep an eye. Otherwise, if I look here, I think it was mod users key clock. 561 00:47:54.340 --> 00:47:58.749 Ingolf Kuss: First of all, what is this? What time is this? 1721. 562 00:47:59.300 --> 00:48:03.109 Ingolf Kuss: It always shows 1721, this is broken, you know? 563 00:48:03.850 --> 00:48:05.550 Ingolf Kuss: And, then. 564 00:48:06.030 --> 00:48:07.809 Florian Kreft (LRZ): Oh, that's your metrics. 565 00:48:07.920 --> 00:48:10.159 Florian Kreft (LRZ): Things that's broken then. 566 00:48:11.110 --> 00:48:11.820 Ingolf Kuss: fixing. 567 00:48:11.820 --> 00:48:15.380 Florian Gleixner: Your metric server is not, not updating? 568 00:48:15.550 --> 00:48:16.180 Florian Kreft (LRZ): Yep. 569 00:48:16.460 --> 00:48:25.770 Ingolf Kuss: What is a metric server? Anyway, and if I go to… Support, what users… clock. 570 00:48:26.320 --> 00:48:38.189 Ingolf Kuss: It should show me here the usage, and this is always… yeah, it's always 0.006. Yeah, but also when I look during the, 571 00:48:38.900 --> 00:48:45.920 Ingolf Kuss: Entitlement, I never get to see a peak here. So why is it scaling, and how many… CPU doesn't. 572 00:48:45.920 --> 00:48:46.340 Florian Kreft (LRZ): I mean. 573 00:48:46.340 --> 00:48:47.119 Ingolf Kuss: I don't know. 574 00:48:47.120 --> 00:48:57.719 Florian Kreft (LRZ): I mean, probably mod users, KeyCloud doesn't actually have to do much with the CPU. It's, like, the most you're probably waiting on here is something like networking requests. 575 00:48:58.020 --> 00:49:03.060 Mahrad Zoonematkermani: I would say it actually does have a load on the CPU, but it's so short. 576 00:49:03.560 --> 00:49:04.450 Florian Kreft (LRZ): Wow, yeah. 577 00:49:04.780 --> 00:49:10.169 Ingolf Kuss: What do you mean with matrix? How do I repair this? This would be helpful to see. 578 00:49:10.430 --> 00:49:14.419 Florian Gleixner: Yeah, did you install a metric server some… someday? 579 00:49:14.420 --> 00:49:17.790 Ingolf Kuss: What is a metric? So, no, volumetric server. 580 00:49:18.660 --> 00:49:19.270 Florian Kreft (LRZ): I'll move into… 581 00:49:19.270 --> 00:49:26.739 Shelley Doljack: And you're… you're going blind through all this if you don't have… if you don't have a metrics thing. 582 00:49:26.740 --> 00:49:27.470 Florian Kreft (LRZ): Yep. 583 00:49:27.810 --> 00:49:37.700 Shelley Doljack: You don't have any data to analyze to figure out, oh, I need to bump up the resources, this is not working the way I have this deployed. 584 00:49:38.060 --> 00:49:48.289 Florian Kreft (LRZ): And Ingov, you won't find it here, this will definitely be in one of the system's namespaces, so this is not part of all, it's like a Kubernetes metrics server. 585 00:49:48.290 --> 00:49:50.449 Florian Gleixner: Just, just go to all names, places. 586 00:49:50.930 --> 00:49:52.070 Florian Kreft (LRZ): Yeah, totally. 587 00:49:52.070 --> 00:49:53.579 Ingolf Kuss: Jesus, so… okay. 588 00:49:54.430 --> 00:49:55.710 Florian Gleixner: Search for metrics. 589 00:50:00.110 --> 00:50:03.910 Florian Gleixner: metrics server. So, you have a metric server here in… 590 00:50:03.910 --> 00:50:04.490 Ingolf Kuss: Sooner. 591 00:50:04.490 --> 00:50:07.270 Florian Gleixner: In the Rancher… in the Rancher namespace? 592 00:50:07.670 --> 00:50:08.859 Florian Kreft (LRZ): No cube system. 593 00:50:09.180 --> 00:50:10.490 Florian Gleixner: Oh, God. 594 00:50:10.490 --> 00:50:11.780 Ingolf Kuss: What is wrong with this? 595 00:50:12.690 --> 00:50:13.540 Florian Gleixner: Don't know. 596 00:50:14.510 --> 00:50:22.369 Florian Kreft (LRZ): can look at the logs, maybe, of that? I mean, if it's working, it might also be just a dashboard issue, actually. 597 00:50:24.740 --> 00:50:29.490 Florian Kreft (LRZ): But you can just look shortly in the logs if it seems to be doing something. 598 00:50:31.650 --> 00:50:37.150 Florian Kreft (LRZ): That's the dashboard metric scraper, that's the part that should display this correctly? 599 00:50:37.700 --> 00:50:40.050 Florian Kreft (LRZ): But this seems, like, correct, it does… 600 00:50:40.780 --> 00:50:59.809 Mahrad Zoonematkermani: Could it be that the burst is so short that it doesn't hit the scraping interval? Because the metrics that you see are being scraped on periods of time, on a specific interval, and if your burst is between that, you won't even notice it. 601 00:51:00.260 --> 00:51:11.719 Florian Kreft (LRZ): Yeah, and the other thing is that the numbers under the graph can just be, like, a purely UI issue. It's possible that this doesn't actually point… 602 00:51:11.720 --> 00:51:14.039 Ingolf Kuss: Yeah, create a scrape Note. 603 00:51:14.040 --> 00:51:16.050 Florian Kreft (LRZ): Yeah, that's… that's a problem. 604 00:51:16.460 --> 00:51:17.080 Mahrad Zoonematkermani: Yeah. 605 00:51:17.540 --> 00:51:19.070 Florian Kreft (LRZ): It's too sweet. 606 00:51:19.070 --> 00:51:20.240 Ingolf Kuss: grape note. 607 00:51:20.240 --> 00:51:23.460 Florian Gleixner: Your metric server doesn't have access to… 608 00:51:24.250 --> 00:51:27.980 Florian Gleixner: To the node, so maybe you have to reinstall it. 609 00:51:27.980 --> 00:51:31.719 Ingolf Kuss: A node is in server. Yeah, okay. 610 00:51:33.000 --> 00:51:36.220 Ingolf Kuss: Fail to scrape node. What does scrape mean? 611 00:51:36.580 --> 00:51:38.519 Florian Gleixner: It just gets to the tough. 612 00:51:38.520 --> 00:51:39.110 Florian Kreft (LRZ): Nope. 613 00:51:39.600 --> 00:51:47.629 Florian Gleixner: The easiest way would be just to use the metrics server Helm chart and reinstall it. 614 00:51:48.080 --> 00:51:48.650 Florian Kreft (LRZ): Yep. 615 00:51:49.420 --> 00:51:50.120 Florian Gleixner: Oh. 616 00:51:51.410 --> 00:51:56.100 Florian Kreft (LRZ): I mean, the pod itself restarting will probably not help here, but… 617 00:51:56.100 --> 00:52:02.880 Ingolf Kuss: I never, consciously installed a metric server. This is probably part of the dashboard stuff, or… 618 00:52:03.270 --> 00:52:04.980 Florian Kreft (LRZ): No… not… 619 00:52:04.980 --> 00:52:12.250 Florian Gleixner: We installed it with, with, aside, along with the dashboard, but, it's not, not directly… 620 00:52:13.070 --> 00:52:14.800 Florian Gleixner: Part of the dashboard. 621 00:52:15.080 --> 00:52:16.000 Ingolf Kuss: I don't know. 622 00:52:18.730 --> 00:52:21.899 Ingolf Kuss: Why don't I just erase this and then it will reinstall? 623 00:52:22.150 --> 00:52:22.690 Florian Gleixner: Yeah. 624 00:52:23.400 --> 00:52:29.000 Florian Kreft (LRZ): You can definitely delete the pod and see if that just fixes it, which… 625 00:52:29.000 --> 00:52:29.420 Ingolf Kuss: this one. 626 00:52:29.420 --> 00:52:32.369 Florian Kreft (LRZ): I don't think it will, but might… it might… 627 00:52:33.150 --> 00:52:34.070 Ingolf Kuss: Just one, huh? 628 00:52:34.710 --> 00:52:35.840 Florian Kreft (LRZ): Yes. 629 00:52:43.390 --> 00:52:47.639 Mahrad Zoonematkermani: One side question, how strict are we on the timing? 630 00:52:51.700 --> 00:52:52.869 Florian Kreft (LRZ): Usually not very. 631 00:52:52.870 --> 00:52:56.100 Ingolf Kuss: Depends on your… What did the others say? 632 00:52:57.170 --> 00:53:01.259 Ingolf Kuss: Yeah, we usually finish on time. 633 00:53:09.180 --> 00:53:15.020 Florian Kreft (LRZ): Can you maybe look at the logs again, if that fixed it? Just go to the pod, to the new one? 634 00:53:16.170 --> 00:53:18.000 Florian Kreft (LRZ): Well, it's just… 635 00:53:18.770 --> 00:53:22.959 Florian Kreft (LRZ): It was just, the first pod that showed up. 636 00:53:29.810 --> 00:53:36.140 Ingolf Kuss: Oh… oh, no metrics, so… Lumetric server. 637 00:53:37.660 --> 00:53:39.230 Florian Kreft (LRZ): Broke metric storage. 638 00:53:43.370 --> 00:53:44.380 Florian Kreft (LRZ): Interesting. 639 00:53:45.040 --> 00:53:48.180 Florian Kreft (LRZ): Can you go to auto-refresh here? 640 00:53:48.720 --> 00:53:49.829 Florian Kreft (LRZ): Of the logs? 641 00:53:50.810 --> 00:53:53.550 Florian Kreft (LRZ): Maybe it was not completely started up yet, but… 642 00:53:55.990 --> 00:53:58.760 Ingolf Kuss: Usually, it's if you click here on the bottom right. 643 00:53:58.760 --> 00:53:59.790 Florian Kreft (LRZ): Oh, no. 644 00:54:02.530 --> 00:54:07.680 Florian Kreft (LRZ): I'm actually not sure how the logs are supposed to look off the metric server, that would maybe help. 645 00:54:10.660 --> 00:54:12.259 Ingolf Kuss: Oh, it's funny. 646 00:54:27.480 --> 00:54:31.980 Ingolf Kuss: Okay… Who does it work? 647 00:54:32.770 --> 00:54:33.540 Ingolf Kuss: No. 648 00:54:35.750 --> 00:54:43.470 Ingolf Kuss: Maybe I can show one more thing. This is the lock of a… Mot… 649 00:54:43.710 --> 00:54:46.029 Ingolf Kuss: Rolls key clock, and 650 00:54:49.350 --> 00:54:54.600 Ingolf Kuss: This is the last part, this comes up, and then the entitlement fails, so… 651 00:54:56.090 --> 00:54:58.170 Ingolf Kuss: I expect there should be some… 652 00:54:59.280 --> 00:55:01.989 Ingolf Kuss: Errors here, at least? At least there are warnings. 653 00:55:03.540 --> 00:55:06.569 Ingolf Kuss: So, maybe my question, what does this mean? 654 00:55:07.900 --> 00:55:16.350 Ingolf Kuss: User already exists, okay, can be, but post to HTTP users. What is HTTP users? I thought it's called mod users… 655 00:55:16.350 --> 00:55:18.409 Mahrad Zoonematkermani: This is a sidebar thing. 656 00:55:19.520 --> 00:55:23.569 Mahrad Zoonematkermani: Please correct me if I'm wrong for… The Stanford… 657 00:55:23.980 --> 00:55:32.289 Shelley Doljack: Yeah, this is… it's… I think this log is from the sidecar, and it's trying to post to… 658 00:55:32.820 --> 00:55:37.909 Shelley Doljack: HTTP users. I've seen this before, this issue that you're having. 659 00:55:38.080 --> 00:55:40.620 Shelley Doljack: It could be that you don't have… 660 00:55:40.800 --> 00:55:44.959 Shelley Doljack: An environment variable set for, 661 00:55:46.610 --> 00:55:51.719 Shelley Doljack: the module. Which one is this that's failing on entitlement? 662 00:55:53.280 --> 00:55:54.360 Ingolf Kuss: Motivus. 663 00:55:55.000 --> 00:55:55.899 Ingolf Kuss: Big clock. 664 00:55:56.760 --> 00:55:58.180 Shelley Doljack: Mod users, Key Clove? 665 00:55:58.180 --> 00:55:58.750 Ingolf Kuss: York. 666 00:56:00.050 --> 00:56:08.200 Ingolf Kuss: So, no, it's, No, that's one hour ago here, so, P… Followed… 667 00:56:15.480 --> 00:56:16.180 Ingolf Kuss: both. 668 00:56:17.940 --> 00:56:19.399 Ingolf Kuss: This was during entitlement. 669 00:56:19.400 --> 00:56:25.220 Shelley Doljack: Do you… do you have a copy URL set to localhost 8082? 670 00:56:25.500 --> 00:56:31.150 Shelley Doljack: For mod users, Key Cloak, Does the mod user's key cloak? 671 00:56:31.510 --> 00:56:37.090 Shelley Doljack: have an environment variable, so copy underscore URL. 672 00:56:37.260 --> 00:56:38.609 Shelley Doljack: And what is it sent to? 673 00:56:40.730 --> 00:56:41.840 Ingolf Kuss: Yes, I think I have… 674 00:56:41.840 --> 00:56:42.899 Shelley Doljack: Because. 675 00:56:42.900 --> 00:56:44.490 Ingolf Kuss: Correctly, but we can look it up. 676 00:56:57.430 --> 00:57:04.950 Shelley Doljack: And then, also, you're saying… System users enabled false. 677 00:57:06.250 --> 00:57:07.570 Shelley Doljack: Everywhere. 678 00:57:08.930 --> 00:57:12.940 Shelley Doljack: All variations of username, system, username. 679 00:57:12.940 --> 00:57:14.519 Ingolf Kuss: And I see the variables. 680 00:57:17.720 --> 00:57:18.640 Ingolf Kuss: the pot? 681 00:57:19.010 --> 00:57:23.720 Shelley Doljack: Click on, yeah, click on your pod there, and then… 682 00:57:25.750 --> 00:57:33.910 Shelley Doljack: What environment variables is it getting? Do you have? Okay, so system user enabled false in that Java options is good. 683 00:57:34.390 --> 00:57:38.100 Shelley Doljack: Okay, so copy URL, localhost 8082. 684 00:57:39.460 --> 00:57:40.620 Ingolf Kuss: Do you see this? 685 00:57:40.620 --> 00:57:41.189 Shelley Doljack: That's right. 686 00:57:41.190 --> 00:57:42.110 Ingolf Kuss: Oh, darn. 687 00:57:43.140 --> 00:57:44.690 Shelley Doljack: What else? 688 00:57:44.690 --> 00:57:45.500 Ingolf Kuss: not hidden. 689 00:57:45.500 --> 00:57:46.880 Shelley Doljack: is missing. 690 00:57:48.120 --> 00:57:50.680 Shelley Doljack: I've seen this error before. 691 00:57:52.750 --> 00:57:53.880 Shelley Doljack: It's like… 692 00:57:54.570 --> 00:58:01.209 Shelley Doljack: what was it? It was trying to create a mod user's key cloak, and it's like, it already exists? 693 00:58:01.860 --> 00:58:03.660 Shelley Doljack: Yeah. 694 00:58:07.090 --> 00:58:09.800 Ingolf Kuss: That's bec- probably because I have, 695 00:58:10.300 --> 00:58:16.870 Ingolf Kuss: Previous entitlement attempts, and the test created a user and then failed after this, maybe? 696 00:58:17.270 --> 00:58:23.440 Shelley Doljack: So we also… we put… we put these in all of our deployments as well, I'll put in the chat. 697 00:58:23.930 --> 00:58:30.699 Shelley Doljack: Folio system user enabled, value false. System user created, false. System user enabled, false. 698 00:58:30.870 --> 00:58:31.590 Shelley Doljack: We just… 699 00:58:31.970 --> 00:58:40.620 Shelley Doljack: We're just, like, throwing that all at everything, because it's not consistent. The developers didn't… 700 00:58:41.440 --> 00:58:49.330 Shelley Doljack: all the dev teams come up with different things, apparently. Like, yeah, I mean… 701 00:58:49.330 --> 00:58:50.020 Ingolf Kuss: Okay. 702 00:58:50.570 --> 00:58:54.700 Shelley Doljack: If you're… If you're entitling, 703 00:58:55.580 --> 00:58:59.140 Shelley Doljack: And you're using the same, like, KeyCloat database? 704 00:59:00.110 --> 00:59:04.669 Shelley Doljack: From something that was successfully entitled previously? 705 00:59:04.960 --> 00:59:05.540 Shelley Doljack: I mean. 706 00:59:05.540 --> 00:59:06.070 Ingolf Kuss: Yeah. 707 00:59:06.070 --> 00:59:13.389 Shelley Doljack: If it were me, I would have… if I was using the same database, I would have truncate all the tables in KeyCloak and Kong and start over. 708 00:59:14.450 --> 00:59:14.950 Shelley Doljack: I wouldn't… 709 00:59:14.950 --> 00:59:16.499 Ingolf Kuss: There is a keyword informed. 710 00:59:16.630 --> 00:59:18.659 Ingolf Kuss: What do you mean, truncated tables? 711 00:59:19.430 --> 00:59:34.280 Ingolf Kuss: Yeah, but this is really… this could be, because entitlement had worked. It had worked. I had worked for a different tenant, for, DICO 2, I called this, and I had entitled the snapshot version of, 712 00:59:35.880 --> 00:59:50.909 Ingolf Kuss: what is it? For more minimal, platform minimal, and then I wanted to go to the sunflower versions, and it never worked. So yeah, this is one suspicion that I have, since it has worked once, it never works for some other. 713 00:59:51.290 --> 00:59:55.450 Ingolf Kuss: Tenant or some other, application. 714 00:59:59.510 --> 01:00:02.660 Ingolf Kuss: So what did you do? Can you say this again? Truncate. 715 01:00:02.960 --> 01:00:09.489 Shelley Doljack: No, I'm sorry. I… what I was… I was saying, like, if you're reusing the same tenant. 716 01:00:09.800 --> 01:00:11.309 Shelley Doljack: And trying to. 717 01:00:11.310 --> 01:00:12.750 Ingolf Kuss: No, it's not the same tenant. 718 01:00:13.200 --> 01:00:14.160 Shelley Doljack: Okay. 719 01:00:14.680 --> 01:00:18.940 Ingolf Kuss: But it's a different application, so wasn't there a problem that you cannot have. 720 01:00:19.910 --> 01:00:26.159 Shelley Doljack: So what are you… you entitled at Platform Minimal, and now you're entitling what? 721 01:00:27.020 --> 01:00:30.799 Ingolf Kuss: I entitled that Platform Minimal Snapshot Version. 722 01:00:31.800 --> 01:00:38.660 Ingolf Kuss: for… and now I'm entitling it for… Sunflower. 723 01:00:39.150 --> 01:00:39.890 Ingolf Kuss: So. 724 01:00:43.340 --> 01:00:48.770 Shelley Doljack: So, in this same tenant, you've entitled app platform minimal. 725 01:00:48.770 --> 01:00:55.949 Ingolf Kuss: This I have entitled successfully for DICU 2, and now this only thing I want to do, entitle this. 726 01:00:56.460 --> 01:00:58.450 Ingolf Kuss: for HPZ test. 727 01:00:59.240 --> 01:01:05.379 Shelley Doljack: And you're entitling a different version of App Platform Minimal in the same tenon. 728 01:01:05.380 --> 01:01:09.680 Ingolf Kuss: Yes, but I have deleted all the entitlements, I have even deleted all the 729 01:01:09.800 --> 01:01:13.989 Ingolf Kuss: The modules, the old version pods, so… 730 01:01:14.490 --> 01:01:20.750 Shelley Doljack: Well, it… I mean, if we go back to the original error in the log, it said that the user 731 01:01:20.920 --> 01:01:30.139 Shelley Doljack: already exists, so… My suspicion is that it doesn't like the fact that you've entitled at a later. 732 01:01:30.140 --> 01:01:30.579 Ingolf Kuss: As I said. 733 01:01:30.580 --> 01:01:31.230 Shelley Doljack: Superintendent. 734 01:01:31.230 --> 01:01:36.410 Ingolf Kuss: That's the HPZ test. I didn't ever entitle for HPZ test yet, so… 735 01:01:36.870 --> 01:01:41.689 Shelley Doljack: Okay, so this is a different tenant that you're entitling. 736 01:01:41.990 --> 01:01:42.670 Ingolf Kuss: Yes. 737 01:01:43.290 --> 01:01:45.260 Shelley Doljack: sunflower version of our. 738 01:01:45.260 --> 01:01:45.830 Ingolf Kuss: And to… 739 01:01:45.830 --> 01:01:46.620 Shelley Doljack: minimal. 740 01:01:46.620 --> 01:01:51.810 Ingolf Kuss: Maybe… maybe we can also lock in vault. I think everything is okay… hold on, Keycloak. 741 01:01:52.030 --> 01:01:58.260 Ingolf Kuss: involved or key cloak, maybe both. I think everything is okay. It's okay there, but 742 01:01:58.810 --> 01:02:02.729 Ingolf Kuss: Let me see, K, C, 4… That's it. 743 01:02:05.310 --> 01:02:06.160 Ingolf Kuss: Cutman. 744 01:02:06.620 --> 01:02:09.989 Ingolf Kuss: It's not called admin, is it? Called Key Clog? Do you know? 745 01:02:11.900 --> 01:02:13.820 Shelley Doljack: It's whatever you set it to. 746 01:02:15.230 --> 01:02:19.289 Shelley Doljack: Maybe admin and secret password? 747 01:02:19.290 --> 01:02:21.899 Ingolf Kuss: No, no, the passport, I know. 748 01:02:26.280 --> 01:02:28.750 minus: When you do an entitlement. 749 01:02:28.900 --> 01:02:38.979 minus: Do you use the same UID for the user, or is it always created with a new UUID? 750 01:02:42.340 --> 01:02:43.540 Ingolf Kuss: For what user? 751 01:02:44.210 --> 01:02:45.710 Ingolf Kuss: Doesn't have a user. 752 01:02:46.010 --> 01:02:48.619 Shelley Doljack: Which user are you asking about, Magnus? 753 01:02:49.740 --> 01:02:56.580 minus: The error message said it tried to… In fact, mod user. 754 01:02:56.760 --> 01:02:59.910 minus: And it says that this user already exists. 755 01:03:00.440 --> 01:03:02.070 Mahrad Zoonematkermani: It's true. 756 01:03:02.070 --> 01:03:06.750 Ingolf Kuss: It tries to create HPZ system user, that it was a dozen entitlement. 757 01:03:07.210 --> 01:03:15.010 minus: Yeah, and that user, does it have the same ID as another user already have in the system? 758 01:03:19.000 --> 01:03:24.380 Shelley Doljack: So, if you… yeah, copy that value, what, 878? So… 759 01:03:24.880 --> 01:03:27.069 Shelley Doljack: in Key Cloak, in the master. 760 01:03:27.070 --> 01:03:28.499 Ingolf Kuss: And we look and kick, look. 761 01:03:28.640 --> 01:03:33.790 Ingolf Kuss: And I have an answer from the… sorry to interrupt, from the… 762 01:03:35.230 --> 01:03:39.759 Ingolf Kuss: So this is KI… AI bot, and, it says… 763 01:03:40.440 --> 01:03:44.479 Ingolf Kuss: So this has different realms, okay? Manage realms, so let's go into… 764 01:03:45.710 --> 01:03:46.949 Shelley Doljack: I want the tenant that you're. 765 01:03:46.950 --> 01:03:51.980 Ingolf Kuss: In Master, I have, folio Backend admin client. 766 01:03:52.990 --> 01:03:53.360 Shelley Doljack: Yeah. 767 01:03:53.360 --> 01:04:05.140 Ingolf Kuss: thesis now here, but this is fully a backend admin client. And then I have… but I'm not enabling master, but I enable HPZtest. And in HPZtest, it has an RM, 768 01:04:05.800 --> 01:04:11.560 Ingolf Kuss: it has clients, but it does not have folio backend admin client, but I am pretty sure it does not. 769 01:04:11.560 --> 01:04:13.460 Shelley Doljack: No, it's not supposed to. 770 01:04:14.050 --> 01:04:17.009 Shelley Doljack: Polio Backend admin client is only in your master. 771 01:04:17.010 --> 01:04:17.330 Ingolf Kuss: Yeah. 772 01:04:17.330 --> 01:04:18.050 Shelley Doljack: Quickly. 773 01:04:18.260 --> 01:04:19.120 Ingolf Kuss: Exactly. 774 01:04:20.240 --> 01:04:21.559 Ingolf Kuss: Okay, so… 775 01:04:21.560 --> 01:04:23.650 Shelley Doljack: Tenant, is it… is it H. 776 01:04:23.650 --> 01:04:25.290 Ingolf Kuss: Is it this HPZ test? 777 01:04:25.860 --> 01:04:31.560 Ingolf Kuss: HPZ test, this is the tenant, this is a realm for the tenant, which I want to Enable. 778 01:04:31.560 --> 01:04:36.669 Shelley Doljack: Okay, so click on Users. That client list looks accurate. 779 01:04:37.020 --> 01:04:37.750 Shelley Doljack: So… 780 01:04:37.750 --> 01:04:41.980 Ingolf Kuss: This is the HVZ system user. This is probably just… After… No, it doesn't. 781 01:04:41.980 --> 01:04:51.119 Shelley Doljack: After you entitle at platform minimal, the only users you have in your tenant should be the system user and mod roles key cloak. 782 01:04:52.360 --> 01:04:56.540 Shelley Doljack: If you click on Attributes to, 783 01:04:58.020 --> 01:05:03.850 Shelley Doljack: Yeah, the system user there. Go back to the… click on the system user. 784 01:05:04.190 --> 01:05:05.439 Shelley Doljack: The actual bill. 785 01:05:06.010 --> 01:05:07.090 Shelley Doljack: So that user ID. 786 01:05:07.090 --> 01:05:08.260 Ingolf Kuss: Yeah, it's… 787 01:05:08.260 --> 01:05:13.319 Shelley Doljack: The user ID of that user in the Folio database. 788 01:05:13.690 --> 01:05:17.939 Shelley Doljack: Just a little tidbit there. And that's the one that is in your error log. 789 01:05:19.350 --> 01:05:21.859 Ingolf Kuss: Oh, because it's already there, but . 790 01:05:21.860 --> 01:05:22.690 Shelley Doljack: Right. 791 01:05:23.340 --> 01:05:25.010 Ingolf Kuss: Do you want me to delete it? 792 01:05:26.970 --> 01:05:28.019 Ingolf Kuss: The negative, don't. 793 01:05:28.020 --> 01:05:33.489 Shelley Doljack: I mean, I think… did that error appear the second time you tried to. 794 01:05:33.490 --> 01:05:35.319 Ingolf Kuss: We are, yes, yes, secondhand. 795 01:05:35.320 --> 01:05:35.920 Shelley Doljack: Okay. 796 01:05:35.920 --> 01:05:36.550 Ingolf Kuss: in. 797 01:05:36.550 --> 01:05:40.970 Shelley Doljack: Well, then maybe it's a red herring, but it's trailing the whole thing? 798 01:05:41.210 --> 01:05:41.780 Ingolf Kuss: Yes. 799 01:05:41.890 --> 01:05:42.740 Ingolf Kuss: Oh. 800 01:05:43.050 --> 01:05:43.890 Ingolf Kuss: Okay. 801 01:05:45.920 --> 01:05:50.690 Shelley Doljack: Did you manually create that mod user's KeyCloak user here in KeyCloak? 802 01:05:51.780 --> 01:05:53.289 Shelley Doljack: Go back to your list of users. 803 01:05:53.290 --> 01:05:54.760 Ingolf Kuss: Good question, Shelly. 804 01:05:55.450 --> 01:05:55.940 Shelley Doljack: Because… 805 01:05:55.940 --> 01:05:57.850 Ingolf Kuss: can be… Wee! 806 01:05:57.850 --> 01:06:00.710 Shelley Doljack: Did you manually create it? Because you don't have to. 807 01:06:01.810 --> 01:06:02.190 Ingolf Kuss: The good. 808 01:06:02.190 --> 01:06:07.250 Shelley Doljack: And you shouldn't have to for Sunflower CSP4. 809 01:06:07.470 --> 01:06:14.890 Shelley Doljack: After you entitle at platform minimal, you're only gonna get the system user and that mod rolls key cloak user. 810 01:06:15.620 --> 01:06:19.029 Shelley Doljack: And you don't need mod users key cloak? 811 01:06:19.710 --> 01:06:20.500 Shelley Doljack: As a kid. 812 01:06:20.500 --> 01:06:21.939 Ingolf Kuss: You want me to delete it? 813 01:06:23.430 --> 01:06:27.669 Ingolf Kuss: Can we look at the other… where it worked? It has worked for DICU, too. 814 01:06:27.990 --> 01:06:29.170 Ingolf Kuss: Okay. Cool. 815 01:06:30.230 --> 01:06:32.200 Ingolf Kuss: Yeah, but it has more users key clock. 816 01:06:32.200 --> 01:06:34.920 Shelley Doljack: You might have manually created it based on… 817 01:06:35.170 --> 01:06:38.069 Shelley Doljack: faulty information that Josh and I gave. 818 01:06:38.070 --> 01:06:39.379 Ingolf Kuss: Yeah, yeah, oh. 819 01:06:41.910 --> 01:06:55.390 Shelley Doljack: But I can confirm that, when we did this for stage, we looked extra special whether or not we needed to, and we didn't manually create anything in KeyCloak. 820 01:06:55.550 --> 01:06:59.789 Shelley Doljack: Before moving on and entitling, App Platform Complete. 821 01:07:02.530 --> 01:07:07.419 Ingolf Kuss: I don't know, I don't see any wrong or missing inconsistent data here, actually. 822 01:07:07.420 --> 01:07:10.280 Shelley Doljack: Yeah, I wouldn't mess with anything in the client. 823 01:07:10.520 --> 01:07:10.755 Ingolf Kuss: Bill. 824 01:07:10.990 --> 01:07:11.730 Shelley Doljack: list. 825 01:07:13.290 --> 01:07:15.029 Ingolf Kuss: Thinking that this is all fine. 826 01:07:16.320 --> 01:07:18.370 Ingolf Kuss: Ugh, yeah, then what's the problem? 827 01:07:18.370 --> 01:07:19.070 Shelley Doljack: I mean, I don't. 828 01:07:19.070 --> 01:07:19.670 Ingolf Kuss: or… 829 01:07:20.150 --> 01:07:25.590 Shelley Doljack: I would go and, like… Maybe… 830 01:07:26.460 --> 01:07:32.280 Shelley Doljack: I don't know. The log that you have, the log message says that the user already exists. 831 01:07:32.280 --> 01:07:35.920 Ingolf Kuss: Exactly, yeah, this is just… so this is just a warning. 832 01:07:36.260 --> 01:07:39.440 Ingolf Kuss: And, yeah, I just… I just was looking at the… 833 01:07:41.460 --> 01:07:43.550 Ingolf Kuss: And this is what… Here's… 834 01:07:43.710 --> 01:07:47.740 Shelley Doljack: But the question is, is that the thing that's making the whole entitlement daily? 835 01:07:47.740 --> 01:07:56.240 Ingolf Kuss: No, yeah, I can… yeah, this is… see, this is the error of… no, the error of the title manager's value for 100, 836 01:07:56.520 --> 01:08:00.529 Ingolf Kuss: Message could not create admin, so what admin do you mean? 837 01:08:00.530 --> 01:08:00.990 Shelley Doljack: Could not. 838 01:08:00.990 --> 01:08:10.159 Ingolf Kuss: Illegal state or exception. What state is illegal? What do you mean? It doesn't say. And, how can I find out? What does this error mean? 839 01:08:10.470 --> 01:08:13.749 Shelley Doljack: Well, where did you grab this log message from? 840 01:08:13.750 --> 01:08:18.840 Ingolf Kuss: Oh, this comes on the console here, and when I do curl on the… Consoles. 841 01:08:18.840 --> 01:08:22.600 Shelley Doljack: So… This is, like, from manager, tenant, and title. 842 01:08:22.609 --> 01:08:26.989 Ingolf Kuss: So what I was suspecting, but it's just a suspicion, it's from what… 843 01:08:27.899 --> 01:08:30.209 Ingolf Kuss: Oh, it's for mod lock and key clock. 844 01:08:30.210 --> 01:08:32.850 Shelley Doljack: Let's look at your mod lock. 845 01:08:32.859 --> 01:08:35.499 Ingolf Kuss: Yeah, let's look on Mocking Kiko, right? 846 01:08:35.500 --> 01:08:36.160 Shelley Doljack: one where. 847 01:08:36.160 --> 01:08:38.059 Ingolf Kuss: Yeah, yeah, yeah, yeah, you're right. 848 01:08:38.060 --> 01:08:39.310 Shelley Doljack: Create a Key Cloak user. 849 01:08:39.310 --> 01:08:43.690 Ingolf Kuss: Why did I look at what users key cloak? No, let's look at… let's do this. 850 01:08:43.979 --> 01:08:47.609 Ingolf Kuss: By the way, it's 8 minutes after the houses, if you want to… 851 01:08:47.990 --> 01:08:50.419 Ingolf Kuss: Not interested in this, and 852 01:08:51.200 --> 01:08:56.689 Ingolf Kuss: you can do some other things. We meet in two weeks again, and then we will talk about the, 853 01:08:57.520 --> 01:08:58.390 Ingolf Kuss: No. 854 01:08:58.930 --> 01:09:04.389 Shelley Doljack: I guess, yeah, review mod logging, key cloak, and make sure that it also has. 855 01:09:05.729 --> 01:09:06.189 Ingolf Kuss: I will do. 856 01:09:06.189 --> 01:09:14.069 Shelley Doljack: Folio system user enabled equals false. System user created false. System user enabled false. 857 01:09:14.350 --> 01:09:17.690 Ingolf Kuss: I want to find this error message in the log, you know? 858 01:09:17.870 --> 01:09:18.939 Ingolf Kuss: So… 859 01:09:19.910 --> 01:09:26.470 Ingolf Kuss: This is what would be another question. How can I see the complete log? How can I get this into an error, an editor? 860 01:09:27.020 --> 01:09:31.200 Ingolf Kuss: So what I would do now, here, is I copy the name of the board. 861 01:09:32.420 --> 01:09:34.369 Ingolf Kuss: If I'm able to do this… 862 01:09:35.720 --> 01:09:38.739 Ingolf Kuss: And I go to my console here. 863 01:09:40.380 --> 01:09:44.420 Ingolf Kuss: And do lock minus follow. 864 01:09:46.240 --> 01:09:48.729 Ingolf Kuss: And it's a long lock, and I don't know… 865 01:09:50.260 --> 01:09:56.449 Ingolf Kuss: Well, how to find the error message? So, 1532 can be, that's, so this is when I… 866 01:09:57.760 --> 01:09:59.370 Ingolf Kuss: Try the entitlement. 867 01:10:02.500 --> 01:10:06.680 Ingolf Kuss: So it tells me 3 times the same, I don't know, 4 times. 868 01:10:07.180 --> 01:10:07.740 Ingolf Kuss: Kind of… 869 01:10:07.740 --> 01:10:10.070 Shelley Doljack: Do you have 3 pods? 870 01:10:10.070 --> 01:10:11.710 Ingolf Kuss: Yeah, yeah. 871 01:10:11.710 --> 01:10:12.720 Shelley Doljack: This is where it becomes. 872 01:10:12.720 --> 01:10:14.389 Ingolf Kuss: It becomes complicated to troubleshoot. 873 01:10:14.390 --> 01:10:18.179 Shelley Doljack: Because now we have to look at three logs. 874 01:10:18.180 --> 01:10:23.300 Ingolf Kuss: Yeah, now it was mode users key clock with reports for mode login key clock. 875 01:10:23.640 --> 01:10:27.210 Ingolf Kuss: I'm not aware of three parts, but there can be. 876 01:10:27.710 --> 01:10:33.559 Ingolf Kuss: So… Okay, where's something? Why does it show me all the… 877 01:10:33.780 --> 01:10:36.179 Ingolf Kuss: System variables time and time again. 878 01:10:36.550 --> 01:10:41.149 Ingolf Kuss: So this is already something very else. 1818, it's probably another day. 879 01:10:41.920 --> 01:10:44.830 Ingolf Kuss: So, no error message. So, there's no error message, just… 880 01:10:45.510 --> 01:10:50.129 Ingolf Kuss: See, this is what I don't understand. I get… The error message here… 881 01:10:51.330 --> 01:10:52.230 Shelley Doljack: Can you look at the… 882 01:10:52.230 --> 01:10:53.060 Ingolf Kuss: concert… 883 01:10:53.060 --> 01:10:55.120 Shelley Doljack: No, I'm. 884 01:10:55.120 --> 01:10:56.599 Ingolf Kuss: So, yeah, yeah, yeah. 885 01:10:56.600 --> 01:10:57.090 Shelley Doljack: our laws. 886 01:10:57.090 --> 01:10:59.930 Ingolf Kuss: On the sidecar, correctly, on the sidecar. 887 01:11:00.080 --> 01:11:02.280 Ingolf Kuss: Where's the sidecar? In the service? 888 01:11:02.280 --> 01:11:08.419 Shelley Doljack: So I'm not sure when you did… I didn't quite see you did that cube control command if you were getting the… 889 01:11:09.660 --> 01:11:12.409 Shelley Doljack: Which container logs you were getting. 890 01:11:12.410 --> 01:11:16.760 Florian Kreft (LRZ): If you click on logs here, you should be able to select the sidecar. 891 01:11:17.100 --> 01:11:18.780 Florian Kreft (LRZ): So if you click on the lock here… 892 01:11:18.780 --> 01:11:20.110 Ingolf Kuss: Cool. Yeah, yeah. 893 01:11:20.140 --> 01:11:20.810 Florian Kreft (LRZ): Nope. 894 01:11:21.960 --> 01:11:23.719 Ingolf Kuss: Click on the locks here. 895 01:11:23.720 --> 01:11:33.670 Florian Kreft (LRZ): Yeah, just open the logs of that pod, and then you have multiple containers in that pod, and one of them is the sidecar, and you can… no, just click on the log button. 896 01:11:35.200 --> 01:11:36.260 Florian Kreft (LRZ): That's fine. 897 01:11:36.430 --> 01:11:38.889 Florian Kreft (LRZ): Just… just open the logs first. 898 01:11:39.300 --> 01:11:42.019 Florian Kreft (LRZ): And then you have a drop-down on the top left. 899 01:11:42.340 --> 01:11:42.870 Ingolf Kuss: Okay. 900 01:11:42.870 --> 01:11:48.950 Florian Kreft (LRZ): Actually… On the top left, on the top, on the top left, you can, yeah. 901 01:11:50.480 --> 01:11:51.070 Florian Kreft (LRZ): Yup. 902 01:11:51.070 --> 01:11:57.620 Ingolf Kuss: I think I can also do minus C sidecar also. So, yeah, that's… here's the 400, yay! 903 01:11:58.390 --> 01:11:59.650 Shelley Doljack: We found it… 904 01:12:00.430 --> 01:12:00.980 Ingolf Kuss: Yeah. 905 01:12:00.980 --> 01:12:05.070 Shelley Doljack: There's lots of 400s. 906 01:12:07.070 --> 01:12:10.270 Mahrad Zoonematkermani: I think that would be for every time it's attempting to do it. 907 01:12:11.720 --> 01:12:12.450 Ingolf Kuss: That's good to know. 908 01:12:12.450 --> 01:12:16.299 Mahrad Zoonematkermani: For multiple endpoints, actually, because one is for… 909 01:12:16.300 --> 01:12:17.090 Ingolf Kuss: Correct. 910 01:12:18.830 --> 01:12:20.219 Mahrad Zoonematkermani: Honestly, host? 911 01:12:20.350 --> 01:12:26.150 Mahrad Zoonematkermani: slash underline slash, the other one is post slash underline slash tenant. 912 01:12:27.220 --> 01:12:28.530 Mahrad Zoonematkermani: Nope. Yep. 913 01:12:30.180 --> 01:12:31.900 Ingolf Kuss: There's no info messages. 914 01:12:32.600 --> 01:12:35.450 Ingolf Kuss: Why does it say 400 as an info message? 915 01:12:39.840 --> 01:12:43.290 Ingolf Kuss: Where's the error message? Jesus, I don't understand… Okay. 916 01:12:43.290 --> 01:12:43.950 Shelley Doljack: There you go. 917 01:12:43.950 --> 01:12:44.470 Ingolf Kuss: Okay. 918 01:12:44.900 --> 01:12:45.819 Ingolf Kuss: Feels look too. 919 01:12:45.820 --> 01:12:48.900 Shelley Doljack: entitlements, Good morning. 920 01:12:49.630 --> 01:12:54.109 Shelley Doljack: So, something's wrong… manager, tenant entitlements is, 921 01:12:54.270 --> 01:12:55.839 Shelley Doljack: Not up and running at this. 922 01:12:55.840 --> 01:12:56.370 Ingolf Kuss: Huh? 923 01:12:56.370 --> 01:12:58.360 Shelley Doljack: And it threw this error. 924 01:12:58.570 --> 01:12:59.620 Shelley Doljack: Maybe? 925 01:13:00.290 --> 01:13:01.230 Shelley Doljack: Maybe yours. 926 01:13:01.230 --> 01:13:04.819 Ingolf Kuss: You're actually right, because it has opened another pot of this, and then… 927 01:13:05.970 --> 01:13:12.479 Shelley Doljack: Yeah, so this could be, like, okay, it's routing traffic to this new pod that's not… 928 01:13:13.100 --> 01:13:17.640 Shelley Doljack: So, get rid of your horizontal pod auto-scaling. 929 01:13:17.750 --> 01:13:25.800 Ingolf Kuss: And it's a lot of work, Shelly. I remember spending half a day redeploying, or even more redeploying, all this with horizontal scaling. 930 01:13:27.130 --> 01:13:29.550 Ingolf Kuss: Okay, how do… 931 01:13:29.550 --> 01:13:43.490 Mahrad Zoonematkermani: And I mean, while you're at it, you can also try to fix the probes that you have. And a good point of that, point of reference for that is, Stanford's, 932 01:13:44.400 --> 01:13:50.640 Mahrad Zoonematkermani: repository. In there, you can actually check For every single module. 933 01:13:51.050 --> 01:13:53.399 Mahrad Zoonematkermani: If it's not the common one? 934 01:13:53.710 --> 01:13:57.019 Mahrad Zoonematkermani: What are the ports, that should be probed? 935 01:13:57.020 --> 01:13:58.130 Ingolf Kuss: All the ports. 936 01:13:58.400 --> 01:14:02.850 Mahrad Zoonematkermani: Yeah, because the ports are… Typically, 8081s. 937 01:14:03.420 --> 01:14:04.809 Mahrad Zoonematkermani: But not always. 938 01:14:04.850 --> 01:14:05.410 Ingolf Kuss: Over the… 939 01:14:05.410 --> 01:14:09.099 Mahrad Zoonematkermani: That's the tricky part. Not all of them are 80-81. 940 01:14:09.780 --> 01:14:15.270 Mahrad Zoonematkermani: You can either see a red light that says this is not ready. 941 01:14:15.270 --> 01:14:17.199 Ingolf Kuss: Just keep mixing notes. Yeah. 942 01:14:17.450 --> 01:14:27.300 Mahrad Zoonematkermani: Or you can actually check their repository, because they already have a values file that overwrites the probes, so… 943 01:14:27.600 --> 01:14:28.940 Mahrad Zoonematkermani: Just fixes the whole thing. 944 01:14:30.990 --> 01:14:31.690 Ingolf Kuss: So, mom… 945 01:14:31.690 --> 01:14:32.130 minus: Yeah, they're… 946 01:14:32.130 --> 01:14:35.199 Ingolf Kuss: Jamie, what do I put here for auto-scaling if I want to do… 947 01:14:35.200 --> 01:14:37.149 Mahrad Zoonematkermani: just through falls. 948 01:14:37.150 --> 01:14:38.310 Shelley Doljack: Oh. 949 01:14:39.920 --> 01:14:41.270 Ingolf Kuss: And the vertex? Okay, I'm. 950 01:14:41.270 --> 01:14:41.999 Shelley Doljack: Unless if you're. 951 01:14:42.000 --> 01:14:44.949 Mahrad Zoonematkermani: No, no, please don't do vertical autoscaler right now. 952 01:14:44.950 --> 01:14:45.820 Ingolf Kuss: Okay. 953 01:14:46.180 --> 01:14:51.049 Shelley Doljack: Since you already are, like, constrained on resources, you're just killing yourself. 954 01:14:51.460 --> 01:14:52.610 Shelley Doljack: I think. 955 01:14:52.610 --> 01:14:53.230 Ingolf Kuss: Okay. 956 01:14:53.980 --> 01:14:56.460 Mahrad Zoonematkermani: And making debugging more complicated. 957 01:14:56.460 --> 01:14:57.500 Ingolf Kuss: Yeah, I never saw… 958 01:14:57.500 --> 01:15:09.189 Mahrad Zoonematkermani: But you can avoid using HPAs at the start. They're very good for proxies, they're very bad for things like databases, or even Java applications. 959 01:15:09.190 --> 01:15:16.979 Ingolf Kuss: I have to become comfortable with it. Could become comfortable with this idea of not setting limits and not scaling, you know. 960 01:15:17.370 --> 01:15:17.800 Mahrad Zoonematkermani: Yes. 961 01:15:17.800 --> 01:15:20.509 Ingolf Kuss: Somehow makes sense, and… 962 01:15:20.870 --> 01:15:28.480 Mahrad Zoonematkermani: I mean, for memory, you might have to do it, because for some of the modules that I was bringing up, I did get OMQ. 963 01:15:28.480 --> 01:15:32.639 Ingolf Kuss: Yeah, no, it didn't, comply about… complain about the memory. 964 01:15:32.640 --> 01:15:33.330 Mahrad Zoonematkermani: zoo. 965 01:15:33.330 --> 01:15:35.359 Ingolf Kuss: CPUs. Yeah. Yeah. 966 01:15:37.160 --> 01:15:41.780 Ingolf Kuss: Yeah, okay, thank you. So, I have some new ideas. 967 01:15:43.630 --> 01:15:47.709 Ingolf Kuss: Some new work. I have to redeploy this all, basically all. 968 01:15:49.460 --> 01:15:54.889 Ingolf Kuss: But that's how it is, you know? Nothing is for free. Oh, you're gonna have to go now. 969 01:15:55.310 --> 01:15:57.250 Ingolf Kuss: See you in two weeks, Shelly. 970 01:15:58.540 --> 01:15:59.440 Ingolf Kuss: Just do the… 971 01:16:00.250 --> 01:16:02.680 minus: Playing around is the only way to learn. 972 01:16:03.740 --> 01:16:05.330 Ingolf Kuss: sorry? 973 01:16:05.590 --> 01:16:07.409 Ingolf Kuss: I will copy the notes. 974 01:16:07.700 --> 01:16:09.119 Ingolf Kuss: Okay, say this again. 975 01:16:09.500 --> 01:16:10.189 Ingolf Kuss: Let's stop today. 976 01:16:10.190 --> 01:16:13.610 minus: The only way to learn is to… Oh. I don't. 977 01:16:13.610 --> 01:16:14.290 Ingolf Kuss: Yeah. 978 01:16:17.510 --> 01:16:18.130 Ingolf Kuss: Yo. 979 01:16:18.130 --> 01:16:25.340 minus: And yeah, isn't it time for you to reset up your Kubernetes cluster from the start? 980 01:16:28.720 --> 01:16:29.750 minus: Nope. 981 01:16:29.910 --> 01:16:31.130 Ingolf Kuss: Yeah? 982 01:16:31.320 --> 01:16:32.730 minus: Yes, please? 983 01:16:34.060 --> 01:16:34.870 minus: Reshut that? 984 01:16:34.870 --> 01:16:43.760 Ingolf Kuss: Kubernetes cluster. I will do this when this… this is a test cluster, and if it works, then I'll set up a production cluster. Before that, I set up nothing. 985 01:16:44.520 --> 01:16:50.360 minus: Yeah, and after you have set up your production cluster, you should remove this cluster. 986 01:16:50.740 --> 01:16:53.820 minus: And restart it from… Bare metal. 987 01:16:55.330 --> 01:16:55.860 Ingolf Kuss: Matthew! 988 01:16:55.860 --> 01:16:57.889 minus: Because it's… it's dirty. 989 01:16:58.570 --> 01:16:59.599 minus: I couldn't have done… 990 01:16:59.600 --> 01:17:02.799 Ingolf Kuss: and metal servers for my IT department. 991 01:17:03.750 --> 01:17:08.250 minus: No, but you could delete the old cluster and start a new one. 992 01:17:10.910 --> 01:17:11.480 Ingolf Kuss: Yeah. 993 01:17:11.950 --> 01:17:14.380 minus: Because you have… It's a big story. 994 01:17:14.380 --> 01:17:15.510 Ingolf Kuss: Ocean, Margos. 995 01:17:15.510 --> 01:17:16.410 minus: Trust her by… 996 01:17:16.410 --> 01:17:19.349 Ingolf Kuss: It's the big solution to start all over again, and… 997 01:17:20.320 --> 01:17:26.699 Ingolf Kuss: But then it will happen that you end up in the same era and say, oh, why did I spend two weeks of time 998 01:17:26.990 --> 01:17:27.960 Ingolf Kuss: So… Yep. 999 01:17:27.960 --> 01:17:47.880 Florian Kreft (LRZ): I think Magnus' point is that you experimented quite a bit, and it's probably easier once you have everything in some kind of working order to just redo that. We also, like, tore down our first Kubernetes cluster completely. 1000 01:17:48.310 --> 01:17:50.430 Ingolf Kuss: I don't have the short time for this. 1001 01:17:50.780 --> 01:17:59.009 Florian Kreft (LRZ): I mean, in theory, if you have documented the setup right, the second installation is way faster than the first one, because you know so much more. 1002 01:17:59.720 --> 01:18:02.499 minus: Even if you don't have documentation, it's much faster. 1003 01:18:02.500 --> 01:18:04.969 Florian Kreft (LRZ): Yeah, you know way more. 1004 01:18:07.630 --> 01:18:10.509 Ingolf Kuss: I don't like to work in this way. 1005 01:18:11.600 --> 01:18:19.589 minus: Yeah, but then you have Put a stone behind you that you need to drag. 1006 01:18:20.930 --> 01:18:26.929 minus: Because you have so much old configuration that doesn't work inside your cluster. 1007 01:18:29.050 --> 01:18:29.670 Mahrad Zoonematkermani: I mean… 1008 01:18:29.930 --> 01:18:30.860 minus: you don't. 1009 01:18:31.600 --> 01:18:37.579 Mahrad Zoonematkermani: If I'm being honest with you, the way I approach this, of course, since I'm using a cluster that 1010 01:18:38.190 --> 01:18:40.479 Mahrad Zoonematkermani: That is being used for other stuff. 1011 01:18:41.580 --> 01:18:48.930 Mahrad Zoonematkermani: until I got here for the past 3 weeks or so, I have torn down the namespace I was working in. 1012 01:18:49.540 --> 01:18:51.449 Mahrad Zoonematkermani: About 5 times. 1013 01:18:51.820 --> 01:18:55.190 Mahrad Zoonematkermani: Complete erasure and start over. 1014 01:18:55.780 --> 01:19:06.339 Mahrad Zoonematkermani: And the key to it is that I have everything I do in pre-written scripts or steps in a Markdown file, and I just rerun them again. 1015 01:19:07.270 --> 01:19:10.170 Mahrad Zoonematkermani: The hard part is the waiting, and… 1016 01:19:10.700 --> 01:19:16.230 Mahrad Zoonematkermani: At this point, if I want to do everything one more time, it's gonna take less than 1 hour. 1017 01:19:16.420 --> 01:19:18.049 Mahrad Zoonematkermani: Maybe 30 minutes. 1018 01:19:19.120 --> 01:19:22.799 Mahrad Zoonematkermani: That doesn't include, of course, tearing down the cluster, but… 1019 01:19:22.940 --> 01:19:28.850 Mahrad Zoonematkermani: If you're using K3S, it should be also very easy. And it's automatable. 1020 01:19:29.230 --> 01:19:33.859 Mahrad Zoonematkermani: And, one side note regarding, resource usage. 1021 01:19:34.510 --> 01:19:42.930 Mahrad Zoonematkermani: It turns out that I'm currently using only 86 to 87 gigabytes of memory in my namespace. 1022 01:19:44.220 --> 01:19:45.700 Mahrad Zoonematkermani: For the whole thing to run. 1023 01:19:45.700 --> 01:19:49.350 Ingolf Kuss: No, I believe. I was also expecting that I have enough. 1024 01:19:49.550 --> 01:19:59.950 Mahrad Zoonematkermani: Yeah, so if you don't have a HPA, and you don't add more than one replica for your first try before going prod-ready, you shouldn't 1025 01:20:00.510 --> 01:20:02.560 Mahrad Zoonematkermani: Worry too much about memory, either. 1026 01:20:02.560 --> 01:20:03.470 Ingolf Kuss: No. 1027 01:20:03.770 --> 01:20:07.710 Mahrad Zoonematkermani: only OOM kills, but that's typical in folio. 1028 01:20:07.880 --> 01:20:09.540 Mahrad Zoonematkermani: As far as I've learned. 1029 01:20:11.830 --> 01:20:12.510 Ingolf Kuss: Yeah. 1030 01:20:12.910 --> 01:20:15.430 Ingolf Kuss: Yeah, no, I don't have out of memory. 1031 01:20:15.650 --> 01:20:16.379 Ingolf Kuss: I must. 1032 01:20:16.690 --> 01:20:21.590 Ingolf Kuss: Yeah, okay, it's a lot of work, I don't know, versus… 1033 01:20:22.270 --> 01:20:31.709 minus: You, you will, it will make stuff easier. I also remove and restart all my computers 1034 01:20:32.150 --> 01:20:34.289 minus: Every major version. 1035 01:20:37.010 --> 01:20:37.910 minus: Don't stop. 1036 01:20:37.910 --> 01:20:39.080 Ingolf Kuss: the computers as a new… 1037 01:20:39.080 --> 01:20:39.559 minus: No, no, no, no. 1038 01:20:39.560 --> 01:20:40.900 Ingolf Kuss: Magnosis, after… 1039 01:20:40.900 --> 01:20:46.980 minus: I, I, I mean… Installing an operating system from scratch. 1040 01:20:49.550 --> 01:21:00.189 minus: But we are using not bare metal, we are using VMs, so they are a bit easier, but we have automated everything. 1041 01:21:04.530 --> 01:21:06.040 Mahrad Zoonematkermani: It's also more secure. 1042 01:21:06.950 --> 01:21:14.480 Mahrad Zoonematkermani: Yeah. Because if there is something injected on the system, And that way of… 1043 01:21:16.410 --> 01:21:20.009 Mahrad Zoonematkermani: injection of the CV is already gone. 1044 01:21:21.200 --> 01:21:33.540 Mahrad Zoonematkermani: By using a brand new machine, or rather, a fresh installed machine, you lower the chances of that sticking around, that residual security issue is gone. 1045 01:21:34.130 --> 01:21:36.230 Mahrad Zoonematkermani: So… I read this. 1046 01:21:36.230 --> 01:21:41.159 Ingolf Kuss: I think this system is much too complicated to reinstall it over and over again. 1047 01:21:41.450 --> 01:21:56.460 Ingolf Kuss: You can do it with a simple system, but not with folio, with 74 modules, plus sidecars, and whatever, and the whole stack with Elasticsearch and Postgres, and it's much too complicated. 1048 01:21:56.460 --> 01:21:57.790 Mahrad Zoonematkermani: 14, 17… 1049 01:21:57.790 --> 01:21:58.870 Ingolf Kuss: Time and time again. 1050 01:21:59.010 --> 01:22:05.880 Mahrad Zoonematkermani: For me, it's 17 bash scripts. I just run them, wait, make sure everything is green, do a little bit of check. 1051 01:22:06.390 --> 01:22:07.319 Mahrad Zoonematkermani: You have to employ someone. 1052 01:22:07.320 --> 01:22:11.689 Ingolf Kuss: someone like you at PZAT has all this, my dear. 1053 01:22:11.690 --> 01:22:20.209 Mahrad Zoonematkermani: No, no, but all these steps that you're taking, you can turn it into a script, so that's gonna save the day for you. 1054 01:22:21.290 --> 01:22:33.009 Florian Gleixner: But usually, you sometimes will have a cluster that has too much data in there, and you're not allowed to tear it down and rebuild it, because they… 1055 01:22:33.700 --> 01:22:34.130 Ingolf Kuss: Yeah. 1056 01:22:34.130 --> 01:22:41.580 Florian Gleixner: I haven't, have, have test data, or data in the… And… Yeah. 1057 01:22:41.850 --> 01:22:42.240 Ingolf Kuss: But. 1058 01:22:42.240 --> 01:22:48.270 Florian Gleixner: For me, our folio test cluster, if we would tear it down, we had to… 1059 01:22:48.610 --> 01:22:53.159 Florian Gleixner: reinstall… how many nodes do we have? 32 nodes, and . 1060 01:22:53.160 --> 01:22:53.780 Florian Kreft (LRZ): Nope. 1061 01:22:54.310 --> 01:22:58.650 Florian Gleixner: We had to move, how much data are there? 1062 01:22:59.690 --> 01:23:05.790 Florian Gleixner: 50 terabytes… 50 terabytes of data. 1063 01:23:06.430 --> 01:23:10.929 Mahrad Zoonematkermani: Yeah, but we are kind of doing that in every upgrade, and I guess… 1064 01:23:11.100 --> 01:23:17.990 Mahrad Zoonematkermani: For us, the test cluster is not really a test cluster. For us, the dev clusters are what we turn down. 1065 01:23:18.340 --> 01:23:20.240 Mahrad Zoonematkermani: Whenever we feel like it. 1066 01:23:21.180 --> 01:23:22.209 Florian Gleixner: That's it. 1067 01:23:22.210 --> 01:23:30.299 Mahrad Zoonematkermani: Yeah, same goes, I guess, for Engel's case. He can simply turn it down if he likes. 1068 01:23:34.940 --> 01:23:35.790 Ingolf Kuss: It's a brave. 1069 01:23:35.790 --> 01:23:37.080 Mahrad Zoonematkermani: smooth, I agree. 1070 01:23:42.060 --> 01:23:42.780 Ingolf Kuss: No, I won't. 1071 01:23:45.400 --> 01:23:46.060 Ingolf Kuss: Shit. 1072 01:23:48.510 --> 01:23:52.189 minus: But you could at least start by setting up just, 1073 01:23:52.420 --> 01:23:59.299 minus: Kong and the smallest of, The fool your stuff. 1074 01:23:59.710 --> 01:24:04.870 Ingolf Kuss: I think, actually, to reply to your suggestions, if I would do this, I would not… 1075 01:24:05.240 --> 01:24:21.480 Ingolf Kuss: tears the old thing down, because I want to have some reference, and, you know, but I have to set up 3 new machines, and that blows that just explodes our limits here. I already have 32 servers, which I have to administer. I don't want to administer more. 1076 01:24:22.780 --> 01:24:32.209 Ingolf Kuss: I have to apply for them, etc. I don't want to apply for this if I don't have to, you know? It's much too much, too much, and I already have a… 1077 01:24:32.330 --> 01:24:40.169 Ingolf Kuss: 6 double servers for… for the folio with Okapi, that's 12, then I have Minio, 13, then I have, 1078 01:24:40.790 --> 01:24:46.580 Ingolf Kuss: Proxy… 15, if, if they want to have this whole, this, this whole, 1079 01:24:47.030 --> 01:24:50.120 Ingolf Kuss: What's the search thing called? Folio? 1080 01:24:50.320 --> 01:24:51.630 minus: Elasticsearch. 1081 01:24:51.810 --> 01:24:54.180 Ingolf Kuss: No, it was a… I don't know. 1082 01:24:54.180 --> 01:24:54.800 Mahrad Zoonematkermani: search. 1083 01:24:57.630 --> 01:24:59.140 Ingolf Kuss: Eric Gibson. 1084 01:24:59.920 --> 01:25:00.420 Ingolf Kuss: No, no, no. 1085 01:25:01.270 --> 01:25:03.739 minus: Is this one of the volume modules. 1086 01:25:04.890 --> 01:25:05.630 Ingolf Kuss: No. 1087 01:25:07.090 --> 01:25:14.259 Ingolf Kuss: Forget it. So it's 14, and then I have the cluster 17. I don't want to have 17 servers for one client. 1088 01:25:14.490 --> 01:25:21.909 Ingolf Kuss: It's much too much too much, and I don't want to always tear this down and set it up all over again. 1089 01:25:23.060 --> 01:25:24.459 Ingolf Kuss: But, yeah. 1090 01:25:24.670 --> 01:25:29.649 minus: One full cluster for us is 6 VMs. 1091 01:25:31.880 --> 01:25:43.439 minus: And we tier them down and up all the time. All the test clusters are… and the lab clusters are ephemeral, so we delete them when we are done with them. 1092 01:25:46.780 --> 01:25:52.629 minus: And that's quite fast. Usually, they live mostly less than half a year. 1093 01:25:55.200 --> 01:26:01.980 Ingolf Kuss: It deletes a whole… Servers? Do you have everything in some repo, repositories, Git or whatever? 1094 01:26:02.140 --> 01:26:03.960 Ingolf Kuss: Yeah, I don't have this, you know. 1095 01:26:04.220 --> 01:26:06.929 Ingolf Kuss: I don't… I'm not so organized in this way. 1096 01:26:07.390 --> 01:26:09.549 minus: No, but that's what you need. 1097 01:26:10.960 --> 01:26:18.779 minus: You need to have everything in Git, otherwise it doesn't work. Because if you have… because what happens if you die? 1098 01:26:20.180 --> 01:26:21.869 minus: what happens to each other? 1099 01:26:21.870 --> 01:26:26.039 Ingolf Kuss: someone like Margaret, and he will set it up in one hour. 1100 01:26:26.200 --> 01:26:28.010 Ingolf Kuss: Maybe that's the best solution. 1101 01:26:28.010 --> 01:26:34.429 minus: But it usually takes a bit more to learn how to interact with folio. 1102 01:26:37.870 --> 01:26:40.139 Ingolf Kuss: Yeah, okay, thank you, this, 1103 01:26:40.900 --> 01:26:43.700 Ingolf Kuss: Some more goes over into a fun discussion, so… 1104 01:26:43.940 --> 01:26:45.110 minus: Yeah. 1105 01:26:46.450 --> 01:26:56.609 Ingolf Kuss: I could say this, I'm not doing this full-time, but only part-time, and I'm the only person who's occupied with this in my institution, but I'm not saying this. 1106 01:26:57.430 --> 01:26:57.930 minus: Yep. 1107 01:26:58.730 --> 01:26:59.790 Ingolf Kuss: I just. 1108 01:26:59.790 --> 01:27:00.469 minus: But, but it's… 1109 01:27:00.470 --> 01:27:04.340 Ingolf Kuss: I just take your advice, so see how much we can do. 1110 01:27:05.120 --> 01:27:09.769 minus: It's a way of documenting the system also, by having it committed to Git. 1111 01:27:12.700 --> 01:27:13.499 Ingolf Kuss: Oh, I forgot. 1112 01:27:13.500 --> 01:27:13.880 minus: I'm listening. 1113 01:27:13.880 --> 01:27:18.540 Ingolf Kuss: committed to Git, but not all things. I still have a, as you see, my German 1114 01:27:18.680 --> 01:27:21.010 Ingolf Kuss: Installation notes, you know. 1115 01:27:23.100 --> 01:27:26.710 minus: I have all my notes in Git. 1116 01:27:28.340 --> 01:27:32.250 minus: Together with, the rest of fully configuration. 1117 01:27:32.700 --> 01:27:34.049 minus: So now we have… 1118 01:27:34.310 --> 01:27:36.859 Ingolf Kuss: We got the and Git. Is this in Git, so… 1119 01:27:36.860 --> 01:27:39.460 minus: One document for each upgrade. 1120 01:27:42.150 --> 01:27:43.779 minus: with a specific… 1121 01:27:44.070 --> 01:27:44.580 Ingolf Kuss: Okay. 1122 01:27:44.580 --> 01:27:45.880 minus: Items you need. 1123 01:27:46.040 --> 01:27:46.720 minus: Alright. 1124 01:27:47.460 --> 01:27:49.219 Ingolf Kuss: We're losing people, so actually… 1125 01:27:49.220 --> 01:27:49.860 minus: Yeah. 1126 01:27:50.770 --> 01:27:54.680 Ingolf Kuss: See you in two weeks, thank you for the advice, and I see how much I can, 1127 01:27:55.160 --> 01:27:58.599 Ingolf Kuss: Do… it's my system at the end, so you're just doing… 1128 01:27:58.710 --> 01:28:01.080 Ingolf Kuss: Meaning it's good to me, so thank you. 1129 01:28:01.500 --> 01:28:02.000 minus: Yep. 1130 01:28:02.830 --> 01:28:04.320 Ingolf Kuss: Bye. Bye.