You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Tue Sep 08, 2026 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
294442240165,24sleep20-21swapper/219:05:072
311632234162,59sleep00-21swapper/019:06:540
311492234164,21sleep10-21swapper/119:06:431
313072205168,24sleep30-21swapper/319:08:473
2169121390,2sleep23149499cyclictest21:10:242
3100721240,3sleep10-21swapper/100:00:151
31494997673,2cyclictest0-21swapper/222:31:162
139932610,1sleep20-21swapper/220:55:112
60202500,2sleep230-21rcuc/221:50:132
88922460,1sleep10-21swapper/123:10:191
133252420,2sleep20-21swapper/222:05:232
31487993411,11cyclictest25194-21tr20:05:201
31487993411,11cyclictest25194-21tr20:05:201
3148799311,17cyclictest0-21swapper/121:01:371
3148799311,12cyclictest15182-21perf19:45:001
3148799294,12cyclictest1584-21switchtime00:05:231
31487992914,3cyclictest0-21swapper/121:40:241
3148799291,12cyclictest30353-21sed00:00:021
3148799291,10cyclictest23136-21unixbench-2d23:40:261
127412290,7sleep011-21rcuc/023:20:020
31494992810,9cyclictest0-21swapper/222:00:112
3148799280,16cyclictest24477-21chrt21:18:421
31478992815,5cyclictest0-21swapper/021:48:200
3147899281,20cyclictest0-21swapper/021:53:180
3147899280,22cyclictest20528-21kworker/u8:000:15:010
31487992710,3cyclictest12707-21latency_hist23:20:011
3147899270,21cyclictest0-21swapper/021:31:530
248282270,3sleep2311ktimersoftd/223:45:192
3149899269,14cyclictest32214-21sshd00:02:173
31478992615,3cyclictest0-21swapper/022:40:320
3147899261,19cyclictest0-21swapper/023:05:110
3147899260,20cyclictest0-21swapper/022:59:430
3147899260,20cyclictest0-21swapper/000:15:200
3147899260,19cyclictest0-21swapper/023:27:410
3147899260,19cyclictest0-21swapper/022:10:230
31498992517,2cyclictest0-21swapper/300:29:173
3149899251,22cyclictest0-21swapper/321:06:333
3149499251,3cyclictest87350irq/126-enp1s0-22:20:212
31487992518,5cyclictest21690-21swap21:10:241
3148799251,4cyclictest10954-21memory22:00:171
3148799251,21cyclictest8102-21processes00:20:241
3147899254,13cyclictest23693-21kworker/u8:221:20:500
3147899253,15cyclictest0-21swapper/021:16:420
31478992520,3cyclictest808-21snmpd19:40:130
3147899251,6cyclictest0-21swapper/022:21:040
3147899251,5cyclictest756-21kworker/u8:021:57:030
3147899251,5cyclictest0-21swapper/023:39:010
3147899251,5cyclictest0-21swapper/023:39:010
31478992515,3cyclictest0-21swapper/000:05:100
3147899251,19cyclictest0-21swapper/000:31:330
3147899251,18cyclictest0-21swapper/023:21:550
3147899251,17cyclictest0-21swapper/000:36:330
3147899250,6cyclictest0-21swapper/023:59:150
3147899250,18cyclictest0-21swapper/000:20:500
3149899249,3cyclictest14447-21/usr/sbin/munin22:10:143
3149899241,21cyclictest0-21swapper/320:45:153
3149899241,21cyclictest0-21swapper/320:25:203
3149899241,11cyclictest0-21swapper/321:47:173
3149899241,10cyclictest0-21swapper/319:40:103
3149499247,5cyclictest21780-21awk20:00:012
31494992418,4cyclictest32058-21head21:35:172
3148799241,20cyclictest6446-21taskset00:17:111
3147899245,9cyclictest23957-21cut23:45:130
3147899242,14cyclictest0-21swapper/022:32:030
3147899241,5cyclictest763-21dbus-daemon20:00:010
3147899241,5cyclictest0-21swapper/022:28:140
3147899241,5cyclictest0-21swapper/022:28:140
3147899241,17cyclictest0-21swapper/022:36:080
3147899241,11cyclictest28386-21perf20:15:010
31498992316,4cyclictest15730-21head23:25:163
31498992315,5cyclictest32486-21ntp_states21:35:233
3149899231,15cyclictest0-21swapper/320:01:033
3149899231,11cyclictest3770-21sshd22:59:173
31498992310,3cyclictest827-21runrttasks00:09:203
3149499239,3cyclictest26236-21cstates23:50:172
3149499232,11cyclictest23627-21munin-run23:45:012
3149499231,20cyclictest0-21swapper/221:19:272
3148799236,3cyclictest0-21swapper/121:28:511
3148799232,3cyclictest0-21swapper/123:46:001
3148799231,8cyclictest0-21swapper/123:20:581
31487992317,3cyclictest5887-21grep19:20:241
3148799231,19cyclictest0-21swapper/122:20:121
3148799231,19cyclictest0-21swapper/120:10:161
3147899236,3cyclictest87550irq/128-enp1s0-20:15:570
3147899233,10cyclictest0-21swapper/022:15:200
3147899232,4cyclictest1848-21grep21:40:160
31478992319,3cyclictest0-21swapper/021:07:180
3147899231,4cyclictest0-21swapper/023:00:540
3147899231,4cyclictest0-21swapper/022:04:300
31478992314,2cyclictest0-21swapper/022:49:450
31478992312,3cyclictest808-21snmpd19:50:150
3147899231,15cyclictest0-21swapper/022:53:150
3147899231,14cyclictest0-21swapper/023:41:220
3147899231,13cyclictest0-21swapper/023:51:240
3147899231,11cyclictest20192-21diskmemload23:10:200
3147899230,14cyclictest0-21swapper/021:10:260
3149899223,4cyclictest0-21swapper/321:52:353
3149899221,9cyclictest0-21swapper/319:47:163
3149899221,3cyclictest87350irq/126-enp1s0-23:20:013
3149899220,20cyclictest0-21swapper/320:40:243
3149899220,20cyclictest0-21swapper/319:15:153
3149899220,20cyclictest0-21swapper/319:10:183
3149499222,7cyclictest7754-21kernelversion00:20:182
3149499222,3cyclictest87650irq/129-enp1s0-21:23:132
3149499221,6cyclictest28913-21sshd21:29:172
3149499221,5cyclictest13382-21grep23:20:152
3149499221,3cyclictest87650irq/129-enp1s0-21:32:402
3149499221,12cyclictest4387-21cat19:20:022
3149499221,10cyclictest3008-40dnf21:42:242
3148799222,18cyclictest0-21swapper/123:52:521
3148799222,18cyclictest0-21swapper/122:46:471
3148799222,18cyclictest0-21swapper/121:20:361
3148799222,18cyclictest0-21swapper/119:31:021
3148799221,8cyclictest0-21swapper/100:28:451
3148799221,3cyclictest0-21swapper/120:26:141
3148799221,3cyclictest0-21swapper/119:29:121
3148799221,20cyclictest0-21swapper/119:18:121
3148799221,18cyclictest20192-21diskmemload22:39:581
3148799221,10cyclictest21778-21perf20:00:011
3148799220,7cyclictest0-21swapper/122:40:531
3147899227,4cyclictest31878-21timerwakeupswit00:00:260
3147899225,10cyclictest15749-21diskstats19:45:140
3147899221,2cyclictest0-21swapper/020:30:200
3147899221,16cyclictest0-21swapper/000:28:230
3147899221,15cyclictest0-21swapper/021:35:210
3147899221,13cyclictest0-21swapper/023:30:400
3147899220,4cyclictest0-21swapper/022:06:500
3149899213,8cyclictest87650irq/129-enp1s0-22:35:203
3149899211,7cyclictest20192-21diskmemload22:15:173
3149899211,17cyclictest13818-21/usr/sbin/munin20:55:223
3149899211,12cyclictest0-21swapper/323:36:253
3149899211,12cyclictest0-21swapper/323:36:253
3149899211,10cyclictest0-21swapper/300:10:223
31498992110,4cyclictest30790-21/usr/sbin/munin20:20:223
3149499212,10cyclictest5939-21chrt23:03:462
3149499211,6cyclictest30504-21sshd22:45:172
3149499211,6cyclictest15061-21/usr/sbin/munin23:25:162
3149499211,12cyclictest0-21swapper/223:35:162
3149499211,12cyclictest0-21swapper/223:35:162
3149499211,11cyclictest22561-21awk22:25:262
3149499211,11cyclictest22561-21awk22:25:262
3149499211,11cyclictest0-21swapper/222:53:182
3148799216,12cyclictest17699-21ntpq22:15:201
3148799212,17cyclictest0-21swapper/123:08:041
3148799212,17cyclictest0-21swapper/122:28:271
3148799212,17cyclictest0-21swapper/122:28:271
3148799212,17cyclictest0-21swapper/120:37:301
3148799212,17cyclictest0-21swapper/119:13:561
3148799211,9cyclictest0-21swapper/123:35:451
3148799211,9cyclictest0-21swapper/123:35:451
31487992117,3cyclictest0-21swapper/122:30:231
3148799211,2cyclictest20192-21diskmemload00:10:121
3148799211,2cyclictest0-21swapper/122:13:491
3148799211,2cyclictest0-21swapper/121:55:411
3148799211,19cyclictest0-21swapper/119:38:351
3148799211,18cyclictest0-21swapper/120:04:061
3148799211,18cyclictest0-21swapper/100:35:431
3148799211,18cyclictest0-21swapper/100:31:301
3148799210,2cyclictest0-21swapper/121:31:171
3147899215,8cyclictest808-21snmpd21:00:300
3147899215,8cyclictest808-21snmpd19:35:130
3147899214,9cyclictest31956-21proc_pri20:20:220
3147899212,9cyclictest539-21/usr/sbin/munin20:25:210
31478992118,2cyclictest0-21swapper/021:25:200
3147899211,12cyclictest11618-21/usr/sbin/munin20:50:230
3147899211,11cyclictest8815-21/usr/sbin/munin19:30:200
3147899211,11cyclictest24204-21/usr/sbin/munin20:05:130
3147899211,11cyclictest24204-21/usr/sbin/munin20:05:130
31478992110,2cyclictest22806-21sshd20:00:170
3149899208,3cyclictest87650irq/129-enp1s0-00:38:393
3149899206,3cyclictest5750irq/9-acpi00:20:153
3149899206,3cyclictest13524-21sshd20:54:173
3149899205,5cyclictest0-21swapper/322:05:243
3149899203,3cyclictest87350irq/126-enp1s0-21:22:183
3149899201,7cyclictest4070-21/usr/sbin/munin23:00:153
3149899201,6cyclictest4712-21/usr/sbin/munin00:15:163
3149899201,6cyclictest29455-40dnf22:43:183
3149899201,6cyclictest11643-21df00:30:163
3149899201,16cyclictest22622-21/usr/sbin/munin21:15:233
3149899200,18cyclictest0-21swapper/319:20:133
3149899200,10cyclictest0-21swapper/323:08:173
3149499206,3cyclictest267682fschecks_count20:10:152
3149499203,3cyclictest87650irq/129-enp1s0-21:00:212
3149499202,3cyclictest87350irq/126-enp1s0-00:25:192
3149499201,6cyclictest11007-21/usr/sbin/munin19:35:122
3149499201,6cyclictest0-21swapper/223:18:392
3149499201,6cyclictest0-21swapper/223:06:082
3149499201,6cyclictest0-21swapper/222:56:342
3149499201,5cyclictest14567-21ntp_states00:35:222
31494992014,3cyclictest25670-21grep20:05:262
31494992014,3cyclictest25670-21grep20:05:262
3149499201,11cyclictest0-21swapper/200:30:592
3148799202,3cyclictest763-21dbus-daemon19:50:171
3148799202,3cyclictest0-21swapper/121:06:311
3148799201,4cyclictest31971-21perl21:35:161
3148799201,18cyclictest0-21swapper/123:25:391
3148799201,18cyclictest0-21swapper/119:46:211
3148799201,17cyclictest0-21swapper/123:04:501
3148799201,17cyclictest0-21swapper/120:17:371
3148799201,16cyclictest0-21swapper/122:53:051
3148799200,18cyclictest0-21swapper/122:55:131
3148799200,18cyclictest0-21swapper/122:08:451
3148799200,18cyclictest0-21swapper/121:51:231
3148799200,18cyclictest0-21swapper/121:46:071
3148799200,18cyclictest0-21swapper/120:33:441
3147899202,6cyclictest4414-21/usr/sbin/munin19:20:230
3147899201,11cyclictest1285-21munin-run19:15:010
3147899201,10cyclictest7224-21/usr/sbin/munin20:40:220
3149899192,11cyclictest0-21swapper/323:22:453
3149899191,7cyclictest26381-21expr23:50:173
3149899191,7cyclictest0-21swapper/322:30:273
3149899191,6cyclictest8659-21irqstats21:55:173
3149899191,6cyclictest0-21swapper/322:25:283
3149899191,6cyclictest0-21swapper/322:25:283
3149899191,6cyclictest0-21swapper/322:20:383
31498991914,3cyclictest25467-21smart_sda20:05:263
31498991914,3cyclictest25467-21smart_sda20:05:263
3149899191,3cyclictest12050irq/123-ahci[0020:00:023
31498991913,3cyclictest9366-21fschecks_time19:30:173
3149899191,2cyclictest0-21swapper/319:35:133
3149899191,16cyclictest0-21swapper/320:30:203
3149899191,12cyclictest0-21swapper/323:15:003
3149899190,9cyclictest0-21swapper/323:45:173
3149499196,3cyclictest10478-21sshd19:31:172
3149499194,3cyclictest87650irq/129-enp1s0-21:55:202
3149499194,3cyclictest12050irq/123-ahci[0019:20:232
3149499193,7cyclictest87350irq/126-enp1s0-00:00:202
3149499191,9cyclictest0-21swapper/200:18:262
3149499191,7cyclictest0-21swapper/222:13:532
3149499191,6cyclictest23972-21latency_hist20:05:012
3149499191,6cyclictest0-21swapper/223:30:322
3149499191,6cyclictest0-21swapper/221:47:282
3149499191,6cyclictest0-21swapper/221:09:192
3149499191,5cyclictest0-21swapper/222:35:522
3149499191,5cyclictest0-21swapper/220:36:502
3149499191,4cyclictest0-21swapper/223:12:002
3149499191,3cyclictest0-21swapper/223:56:462
31494991912,4cyclictest1421-21munin-node19:15:012
3149499191,11cyclictest0-21swapper/222:16:382
3149499191,11cyclictest0-21swapper/200:11:362
3149499191,10cyclictest0-21swapper/222:42:532
3149499191,10cyclictest0-21swapper/200:07:232
3148799191,5cyclictest0-21swapper/120:21:221
3148799191,17cyclictest0-21swapper/120:58:081
3148799191,17cyclictest0-21swapper/120:51:101
3148799191,17cyclictest0-21swapper/120:45:321
3148799190,18cyclictest0-21swapper/123:31:381
3148799190,17cyclictest0-21swapper/120:40:061
3147899194,8cyclictest87550irq/128-enp1s0-20:55:290
3147899191,6cyclictest2222-21/usr/sbin/munin19:15:140
3147899191,11cyclictest5030-21/usr/sbin/munin20:35:180
3147899191,10cyclictest808-21snmpd20:45:280
3147899191,10cyclictest7620-21netstat19:25:200
16272190,3sleep39-21rcu_preempt21:40:153
3149899181,9cyclictest0-21swapper/321:26:163
3149899181,8cyclictest0-21swapper/323:58:253
3149899181,5cyclictest0-21swapper/322:52:253
3149899181,3cyclictest22229-21seq21:13:033
31498991812,3cyclictest6100-21ntpq20:35:233
3149899181,15cyclictest29828-21/usr/sbin/munin22:45:243
3149899180,9cyclictest0-21swapper/322:01:333
3149899180,6cyclictest0-21swapper/320:17:073
3149499182,13cyclictest12749-21cut20:50:212
3149499181,6cyclictest0-21swapper/220:26:362
3149499181,5cyclictest28614-21grep20:15:132
3149499181,5cyclictest14118-21sa119:40:172
3149499180,3cyclictest0-21swapper/220:40:142
3149899173,3cyclictest16383-21awk21:00:173
3149899172,6cyclictest0-21swapper/323:41:183
31498991712,3cyclictest27021-21gltestperf20:10:163
3149899171,13cyclictest7245-21sed19:25:163
31498991711,3cyclictest17602-21/usr/sbin/munin19:50:183
3149899170,16cyclictest0-21swapper/323:31:173
3149499171,5cyclictest8476-21NetworkManager19:28:472
3149499171,5cyclictest17602-21/usr/sbin/munin19:50:132
3149499170,6cyclictest0-21swapper/219:45:132
31498991611,3cyclictest30278-21ntp_offset21:30:243
3149499164,2cyclictest0-21swapper/220:20:202
31494991611,3cyclictest9423-21/usr/sbin/munin20:45:132
3149499159,3cyclictest3639-21interrupts20:30:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional