You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-06 - 09:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Nov 06, 2025 00:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
169472241162,24sleep10-21swapper/119:09:551
168682232157,24sleep00-21swapper/019:08:520
169382205167,25sleep20-21swapper/219:09:472
166252203165,25sleep30-21swapper/319:05:453
1705799103100,2cyclictest0-21swapper/219:18:092
86122580,3sleep121-21rcuc/122:30:001
85902520,2sleep00-21swapper/022:29:550
17057994941,3cyclictest0-21swapper/219:49:552
291692460,5sleep029174-21fschecks_time23:15:160
1706299366,6cyclictest0-21swapper/322:00:163
1706299364,7cyclictest31735-21unixbench_multi22:05:263
17062993615,4cyclictest19816-21unixbench_multi00:05:263
1706299361,13cyclictest0-21swapper/300:12:133
1706299351,6cyclictest0-21swapper/323:15:573
1706299343,12cyclictest10772-40dnf22:34:453
1706299341,12cyclictest0-21swapper/321:42:313
124522340,3sleep31706299cyclictest20:10:163
1706299331,6cyclictest0-21swapper/322:18:343
1706299331,14cyclictest0-21swapper/323:20:233
1706299321,13cyclictest0-21swapper/323:05:163
1704499324,13cyclictest2645-21munin-run23:30:000
1706299310,7cyclictest0-21swapper/321:02:313
1704499311,14cyclictest13651-21awk23:55:020
1704499310,23cyclictest14782-21kworker/u8:023:58:200
1706299301,12cyclictest25803-21/usr/sbin/munin21:55:233
1705799301,8cyclictest29161-21kworker/2:000:04:452
1706299291,15cyclictest0-21swapper/319:33:333
1706299290,3cyclictest0-21swapper/322:55:533
1705799291,4cyclictest0-21swapper/222:04:002
1705199293,3cyclictest86850irq/129-enp1s0-21:10:121
1706299281,3cyclictest88602vmstat23:40:263
1705799285,10cyclictest0-21swapper/222:58:252
1704499280,26cyclictest0-21swapper/022:04:010
46152270,4sleep230-21rcuc/222:20:102
1706299270,10cyclictest0-21swapper/322:40:293
1706299270,10cyclictest0-21swapper/320:30:193
1705799271,6cyclictest0-21swapper/223:30:382
1705799271,6cyclictest0-21swapper/221:45:062
1705799271,4cyclictest0-21swapper/223:28:292
17051992724,2cyclictest0-21swapper/122:17:081
17051992724,2cyclictest0-21swapper/119:35:261
17051992724,2cyclictest0-21swapper/100:18:001
1705199272,22cyclictest7040-21latency_hist23:40:011
1705199271,4cyclictest0-21swapper/100:20:361
1704499270,24cyclictest0-21swapper/000:15:190
1706299261,3cyclictest29996-21ntpq19:35:193
1706299261,10cyclictest0-21swapper/319:24:063
1706299261,10cyclictest0-21swapper/319:24:053
1706299260,9cyclictest0-21swapper/320:50:593
1706299260,10cyclictest0-21swapper/319:51:533
1705799267,9cyclictest13742-21cut22:40:162
1705799261,6cyclictest0-21swapper/223:56:072
1705799261,6cyclictest0-21swapper/223:01:122
1705799261,6cyclictest0-21swapper/222:45:192
1705799261,5cyclictest0-21swapper/222:34:322
1705799261,5cyclictest0-21swapper/200:11:552
1705799261,4cyclictest0-21swapper/200:15:182
17051992623,2cyclictest0-21swapper/100:13:561
1705199261,9cyclictest24586-21df_inode23:05:161
1704499262,11cyclictest28725-21idleruntime-cro19:35:010
1704499260,24cyclictest0-21swapper/023:44:230
1704499260,23cyclictest0-21swapper/023:36:460
1706299251,10cyclictest0-21swapper/320:59:573
1706299251,10cyclictest0-21swapper/300:39:413
1706299250,9cyclictest0-21swapper/321:14:253
1706299250,9cyclictest0-21swapper/319:40:493
1706299250,7cyclictest0-21swapper/321:16:153
1706299250,12cyclictest0-21swapper/319:15:173
1705799255,9cyclictest0-21swapper/221:26:572
17057992522,2cyclictest0-21swapper/223:13:322
1705799251,6cyclictest0-21swapper/223:36:452
1705799251,5cyclictest20957-21kworker/u8:121:42:282
1705799251,5cyclictest0-21swapper/223:48:262
1705799251,5cyclictest0-21swapper/223:48:262
1705799251,5cyclictest0-21swapper/223:44:222
1705799251,4cyclictest0-21swapper/223:50:252
1705799251,4cyclictest0-21swapper/222:54:112
1705799250,6cyclictest14782-21kworker/u8:000:28:072
1705199253,12cyclictest0-21swapper/122:54:231
1705199252,21cyclictest0-21swapper/122:56:361
17051992521,3cyclictest0-21swapper/122:40:111
1705199251,5cyclictest0-21swapper/122:10:311
1705199251,21cyclictest354-21ntp_states20:55:221
1705199251,21cyclictest30542-21/usr/sbin/munin20:50:201
1705199251,21cyclictest26754-21cpu19:30:121
1705199251,21cyclictest14569-21latency_hist21:30:011
1705199251,20cyclictest23798-21expr21:50:131
1704499250,23cyclictest0-21swapper/021:42:290
1704499250,22cyclictest0-21swapper/023:48:270
1704499250,22cyclictest0-21swapper/023:48:260
1704499250,12cyclictest31544-21awk20:55:020
1706299243,15cyclictest0-21swapper/319:46:333
1706299241,7cyclictest0-21swapper/322:48:113
1706299241,7cyclictest0-21swapper/321:34:213
1706299241,7cyclictest0-21swapper/321:34:213
1706299241,6cyclictest0-21swapper/322:20:443
1706299241,12cyclictest0-21swapper/319:59:173
1706299241,10cyclictest0-21swapper/300:04:133
1706299240,9cyclictest0-21swapper/319:29:253
1706299240,7cyclictest0-21swapper/323:01:463
1706299240,6cyclictest26106-21kworker/u8:223:10:283
1706299240,3cyclictest0-21swapper/323:55:293
1706299240,3cyclictest0-21swapper/321:35:453
1705799244,9cyclictest5777-21diskmemload21:59:522
1705799243,8cyclictest86850irq/129-enp1s0-22:10:242
1705799241,5cyclictest0-21swapper/223:06:472
1705799241,10cyclictest978-21sshd20:55:272
1705199242,19cyclictest7906-21df_inode20:00:151
1705199242,10cyclictest4455-21cpu22:20:091
1705199241,6cyclictest0-21swapper/123:20:541
1705199241,4cyclictest30078-21if_err_enp2s000:30:181
1705199241,21cyclictest0-21swapper/123:25:151
1705199241,21cyclictest0-21swapper/121:17:201
1705199241,21cyclictest0-21swapper/120:15:051
1705199241,20cyclictest23471-21ls23:00:261
1705199241,20cyclictest0-21swapper/119:10:191
1705199241,11cyclictest1057-21cat00:39:551
1705199240,10cyclictest29934-21kworker/u8:022:08:511
1704499248,9cyclictest0-21swapper/022:43:470
17044992418,3cyclictest29291-21/usr/sbin/munin00:30:240
17044992412,3cyclictest0-21swapper/021:50:020
1704499241,20cyclictest0-21swapper/022:07:400
17044992410,3cyclictest4786-21diskstats22:20:120
1704499240,22cyclictest0-21swapper/023:22:360
233832231,3sleep023407-21ntp_states19:20:210
233832231,3sleep023407-21ntp_states19:20:210
1706299233,3cyclictest86750irq/128-enp1s0-20:41:053
1706299233,2cyclictest86550irq/126-enp1s0-22:29:473
1706299231,7cyclictest0-21swapper/321:23:413
1706299231,7cyclictest0-21swapper/321:07:333
1706299231,15cyclictest0-21swapper/320:35:153
1706299231,11cyclictest0-21swapper/322:36:453
1706299231,11cyclictest0-21swapper/322:36:453
1706299231,10cyclictest0-21swapper/300:18:053
1706299230,8cyclictest0-21swapper/322:11:213
1705799234,9cyclictest0-21swapper/221:24:312
1705799234,12cyclictest7143-21sshd19:57:422
1705799233,11cyclictest5777-21diskmemload23:15:222
1705799231,9cyclictest86550irq/126-enp1s0-19:33:332
1705799231,9cyclictest20921-21tr20:30:132
1705799231,6cyclictest13528-21grep20:10:272
1705799231,4cyclictest0-21swapper/223:22:362
1705799231,4cyclictest0-21swapper/200:08:212
1705799231,11cyclictest0-21swapper/222:06:012
1705199231,8cyclictest16793-21if_enp1s000:00:171
1705199231,5cyclictest0-21swapper/121:22:341
1705199231,4cyclictest0-21swapper/123:40:111
17051992312,3cyclictest0-21swapper/119:45:291
1705199231,21cyclictest0-21swapper/121:49:151
1705199231,20cyclictest0-21swapper/121:05:161
1705199231,20cyclictest0-21swapper/121:02:111
1705199230,6cyclictest0-21swapper/123:32:061
1705199230,6cyclictest0-21swapper/122:47:481
1705199230,5cyclictest0-21swapper/122:34:271
1705199230,5cyclictest0-21swapper/100:07:051
1704499239,4cyclictest0-21swapper/021:35:250
17044992316,4cyclictest14974-21irqstats20:15:180
1704499231,20cyclictest0-21swapper/022:34:470
17044992310,9cyclictest17347-21systemd-journal20:20:200
17044992310,7cyclictest0-21swapper/000:35:500
17044992310,7cyclictest0-21swapper/000:13:060
1704499230,6cyclictest0-21swapper/000:04:460
1704499230,21cyclictest0-21swapper/023:33:090
1704499230,21cyclictest0-21swapper/023:06:470
1704499230,21cyclictest0-21swapper/022:54:110
1706299229,3cyclictest0-21swapper/323:48:063
1706299229,3cyclictest0-21swapper/323:48:053
1706299221,6cyclictest0-21swapper/322:50:353
1706299221,15cyclictest0-21swapper/323:27:493
1706299221,10cyclictest0-21swapper/320:00:053
1706299220,8cyclictest0-21swapper/323:31:153
1706299220,7cyclictest86550irq/126-enp1s0-23:50:113
1706299220,7cyclictest0-21swapper/320:27:453
1706299220,7cyclictest0-21swapper/320:27:453
1706299220,11cyclictest0-21swapper/321:54:213
1705799224,8cyclictest86850irq/129-enp1s0-20:45:262
1705799222,7cyclictest24227-21sensors_temp20:35:242
1705799221,8cyclictest17393-21sshd20:20:192
1705799221,8cyclictest0-21swapper/222:39:092
1705799221,8cyclictest0-21swapper/222:39:092
1705799221,7cyclictest5783-21ls21:10:012
1705799221,7cyclictest0-21swapper/221:18:422
1705799221,7cyclictest0-21swapper/200:36:502
1705799221,4cyclictest0-21swapper/221:31:392
1705799221,4cyclictest0-21swapper/221:31:392
1705799221,11cyclictest14634-21snmpd20:05:182
1705799221,11cyclictest0-21swapper/222:27:472
1705799221,11cyclictest0-21swapper/221:14:132
1705799221,10cyclictest0-21swapper/222:15:222
1705199227,3cyclictest12173-21cat20:10:141
1705199223,2cyclictest86850irq/129-enp1s0-23:55:151
1705199222,18cyclictest0-21swapper/123:51:471
1705199221,3cyclictest17436-21ntp_states20:20:211
1705199221,3cyclictest0-21swapper/122:00:241
1705199221,2cyclictest86850irq/129-enp1s0-21:39:141
1705199221,2cyclictest0-21swapper/121:30:161
1705199221,2cyclictest0-21swapper/121:30:161
1705199221,20cyclictest0-21swapper/120:06:161
1705199221,19cyclictest0-21swapper/123:47:301
1705199221,19cyclictest0-21swapper/123:47:301
1705199221,19cyclictest0-21swapper/120:45:361
1705199221,19cyclictest0-21swapper/120:35:541
1705199221,19cyclictest0-21swapper/120:31:061
1705199221,19cyclictest0-21swapper/120:26:421
1705199221,19cyclictest0-21swapper/120:26:421
1705199221,19cyclictest0-21swapper/119:50:501
1705199221,19cyclictest0-21swapper/119:43:011
1705199221,19cyclictest0-21swapper/119:24:551
1705199221,19cyclictest0-21swapper/119:24:551
1705199221,19cyclictest0-21swapper/100:29:371
1705199220,9cyclictest0-21swapper/121:58:551
1705199220,5cyclictest0-21swapper/121:43:111
1704499228,8cyclictest0-21swapper/021:12:220
1704499222,12cyclictest0-21swapper/022:39:160
1704499222,12cyclictest0-21swapper/022:39:160
17044992215,2cyclictest0-21swapper/019:15:010
1704499221,16cyclictest0-21swapper/023:12:500
1704499221,14cyclictest0-21swapper/000:08:170
17044992210,3cyclictest0-21swapper/021:25:150
1704499220,2cyclictest0-21swapper/000:28:080
1706299216,8cyclictest86850irq/129-enp1s0-20:45:253
1706299213,11cyclictest0-21swapper/319:10:253
1706299211,6cyclictest0-21swapper/321:47:233
1706299211,6cyclictest0-21swapper/300:34:303
1706299211,5cyclictest0-21swapper/323:35:133
1706299210,3cyclictest0-21swapper/300:22:233
1705799211,6cyclictest0-21swapper/221:53:352
1705799211,5cyclictest24310-21munin-run19:25:012
1705799211,5cyclictest24310-21munin-run19:25:012
1705799211,5cyclictest0-21swapper/221:36:202
17057992115,3cyclictest27137-21dump-pmu-power20:45:012
1705799211,14cyclictest30552-21smart_sda00:30:252
1705799211,11cyclictest14634-21snmpd21:00:292
1705799211,10cyclictest19911-21kworker/u8:200:21:072
17051992117,3cyclictest0-21swapper/119:15:501
1705199211,4cyclictest7337-21munin-run20:00:011
1705199211,17cyclictest25842-21hddtemp_smartct20:40:181
1705199210,4cyclictest0-21swapper/123:10:121
1704499217,7cyclictest0-21swapper/021:57:200
1704499217,3cyclictest3322-21open_files22:15:240
1704499212,5cyclictest0-21swapper/021:20:140
17044992118,2cyclictest0-21swapper/021:15:210
1704499211,3cyclictest15786-21diskstats22:45:200
17044992113,3cyclictest0-21swapper/000:20:250
1704499211,18cyclictest0-21swapper/021:50:280
1704499211,17cyclictest26342-21proc_pri20:40:260
1704499210,20cyclictest0-21swapper/022:12:460
1706299206,3cyclictest5777-21diskmemload00:29:373
1706299202,8cyclictest5777-21diskmemload21:25:173
1706299202,10cyclictest14165-21/usr/sbin/munin20:15:193
1706299201,14cyclictest17441-21ntp_states20:20:213
1706299200,8cyclictest0-21swapper/320:06:513
1705799204,4cyclictest26468-21sshd19:29:422
1705799201,11cyclictest28934-21/usr/sbin/munin19:35:112
1705799201,10cyclictest86550irq/126-enp1s0-20:50:132
1705799201,10cyclictest0-21swapper/220:26:362
1705799201,10cyclictest0-21swapper/220:26:362
1705799201,10cyclictest0-21swapper/219:53:012
1705199201,4cyclictest24536-21/usr/sbin/munin19:25:181
1705199201,14cyclictest784-21NetworkManager23:16:011
1704499207,3cyclictest1292-21df_inode19:45:150
1704499205,4cyclictest15142-21df_inode21:30:140
1704499205,4cyclictest15142-21df_inode21:30:140
1704499202,15cyclictest6888-21tail19:55:260
1704499201,9cyclictest7564-21/usr/sbin/munin20:00:170
17044992017,2cyclictest0-21swapper/023:02:430
1704499201,5cyclictest4878-21ntp_states21:05:260
17044992011,4cyclictest0-21swapper/019:38:440
1705799195,3cyclictest86750irq/128-enp1s0-20:15:172
1705799192,3cyclictest86550irq/126-enp1s0-19:13:002
1705799191,5cyclictest7902-21df20:00:152
1705799191,5cyclictest0-21swapper/219:40:132
1705199191,15cyclictest12432-21timerandwakeup22:35:251
1705199191,15cyclictest12432-21timerandwakeup22:35:251
1704499199,7cyclictest10103-21df20:05:150
1704499199,3cyclictest22966-21/usr/sbin/munin20:35:260
1704499198,2cyclictest0-21swapper/020:26:440
1704499198,2cyclictest0-21swapper/020:26:440
1704499195,2cyclictest0-21swapper/019:50:260
1704499191,7cyclictest28941-21grep20:45:270
1704499191,6cyclictest24536-21/usr/sbin/munin19:25:260
1704499191,5cyclictest0-21swapper/021:03:230
1704499191,4cyclictest20652-21/usr/sbin/munin22:55:210
1704499191,3cyclictest0-21swapper/020:11:230
217972181,2sleep00-21swapper/020:30:230
1704499188,3cyclictest0-21swapper/020:55:180
17044991813,3cyclictest32112-21needreboot19:40:180
1704499175,2cyclictest21840-21sshd19:16:430
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional