You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Dec 04, 2025 00:44:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1623991080,77rtkit-daemon0-21swapper/219:07:192
5904472850,7sleep3590448-21latency_hist22:20:003
52293827439,28sleep00-21swapper/019:07:100
52283927456,13sleep30-21swapper/319:05:583
52281726844,15sleep10-21swapper/119:05:381
52330299639,16cyclictest1753-21gnome-shell20:35:151
52330299624,39cyclictest612609-21diskstats23:20:181
162399620,9rtkit-daemon1622-21rtkit-daemon23:40:010
523302996012,16cyclictest1753-21gnome-shell20:45:031
52330299587,28cyclictest539742-21latency_hist19:55:021
162399580,8rtkit-daemon1622-21rtkit-daemon23:50:580
6159092570,9chrt615914-21cpuspeed_turbos23:30:130
523304995726,10cyclictest34-21ksoftirqd/223:25:282
52330299576,26cyclictest629714-21munin-run00:10:001
523302995714,17cyclictest619695-21irqstats23:40:231
162399570,8rtkit-daemon1622-21rtkit-daemon23:53:131
523304995620,20cyclictest34-21ksoftirqd/200:00:002
52330299565,31cyclictest634767-21cat00:20:311
52330299565,31cyclictest614340-21diskstats23:25:161
52330299565,27cyclictest596975-21timerwakeupswit22:35:311
52330299564,31cyclictest540718-21sed19:55:311
523302995611,20cyclictest610854-40lsb_release23:15:191
523304995534,5cyclictest141rcu_preempt19:25:232
52330299557,25cyclictest1753-21gnome-shell22:00:021
52330299555,15cyclictest1753-21gnome-shell20:08:421
523302995512,23cyclictest1753-21gnome-shell20:01:551
523302995512,23cyclictest1753-21gnome-shell20:01:551
523302995511,24cyclictest560083-21memory20:50:231
162399550,8rtkit-daemon1622-21rtkit-daemon23:46:531
52330299549,17cyclictest625820-21seq23:56:131
52330299547,11cyclictest1753-21gnome-shell21:00:221
52330299547,11cyclictest1753-21gnome-shell21:00:211
523302995413,22cyclictest630183-21cnf-update-db00:10:031
523302995412,22cyclictest1753-21gnome-shell22:03:351
523302995411,17cyclictest639400-21cat00:35:231
52330299538,17cyclictest1753-21gnome-shell22:45:031
52330299536,14cyclictest592840-21hddtemp_smartct22:25:191
523302995324,25cyclictest626473-21cups-browsed00:00:051
523302995312,16cyclictest577922-21taskset21:40:311
162399530,8rtkit-daemon1622-21rtkit-daemon23:42:150
52330999529,6cyclictest141rcu_preempt23:56:403
523304995229,19cyclictest619296-21/usr/sbin/munin23:40:152
523304995220,18cyclictest34-21ksoftirqd/223:35:012
52330299526,18cyclictest587162-21latency_hist22:10:021
52330299526,15cyclictest527424-21wc19:20:021
52330299526,15cyclictest527424-21wc19:20:021
52330299525,28cyclictest573421-21/usr/sbin/munin21:30:141
52330299523,20cyclictest562245-21sed20:55:311
162399520,45rtkit-daemon3242254-21gmain20:59:400
5841782510,6sleep22197014-21PK-Backend22:00:182
52330499511,9cyclictest2197014-21PK-Backend23:06:162
523304995119,17cyclictest34-21ksoftirqd/219:55:002
523302995110,23cyclictest1753-21gnome-shell21:09:381
523302995110,22cyclictest617105-21runrttasks23:33:551
162399510,5rtkit-daemon2197014-21PK-Backend21:09:563
52330499503,18cyclictest34-21ksoftirqd/223:15:022
523304995018,18cyclictest34-21ksoftirqd/222:55:002
52330299508,19cyclictest1069-21runrttasks19:38:001
52330299508,19cyclictest1069-21runrttasks19:37:591
52330299507,17cyclictest1753-21gnome-shell19:24:511
52330299507,16cyclictest1753-21gnome-shell19:12:371
52330299507,16cyclictest1753-21gnome-shell19:12:371
52330299506,18cyclictest1753-21gnome-shell22:55:291
52330299505,19cyclictest568476-21cstates21:15:151
523309994921,23cyclictest42-21ksoftirqd/321:40:283
523309994921,23cyclictest42-21ksoftirqd/319:55:253
52330299499,35cyclictest632366-21cat00:15:171
52330299498,17cyclictest1753-21gnome-shell20:27:251
52330299497,15cyclictest572117-21irqstats21:25:221
52330299496,27cyclictest1753-21gnome-shell19:30:591
52330299496,17cyclictest606854-21expr23:05:151
52330299496,16cyclictest556207-21irqstats20:40:231
52330299495,19cyclictest576428-21runrttasks21:38:231
52330299494,27cyclictest588987-21grep22:15:171
52330299494,18cyclictest552530-21expr20:30:141
523302994911,14cyclictest571354-21cron21:25:011
523302994911,14cyclictest571354-21cron21:25:001
5577142480,5sleep02197014-21PK-Backend20:45:110
523309994825,19cyclictest42-21ksoftirqd/322:05:453
523304994819,17cyclictest34-21ksoftirqd/222:36:362
523304994813,20cyclictest34-21ksoftirqd/221:00:022
523304994812,18cyclictest34-21ksoftirqd/200:05:012
52330299487,18cyclictest1753-21gnome-shell21:45:151
52330299487,16cyclictest1753-21gnome-shell23:10:401
52330299485,15cyclictest617479-21apt23:35:131
523302994811,12cyclictest1753-21gnome-shell21:50:461
523304994738,5cyclictest331rcuc/200:12:272
523304994717,9cyclictest34-21ksoftirqd/221:40:192
523304994712,14cyclictest34-21ksoftirqd/220:20:012
52330299477,15cyclictest636518-21cat00:25:331
52330299476,17cyclictest602606-40lsb_release22:51:221
52330299473,19cyclictest545199-21cpuspeed_turbos20:10:141
523298994719,24cyclictest1753-21gnome-shell21:08:370
162399470,5rtkit-daemon2197014-21PK-Backend21:31:093
162399470,41rtkit-daemon1775-21gvfs-afc-volume00:13:480
162399470,39rtkit-daemon621813-21python323:45:270
5765902460,7sleep0576580-21munin-run21:40:010
523309994617,25cyclictest610439-21find23:15:123
52330999461,15cyclictest584427-21cut22:00:233
52330499465,18cyclictest2179-21gvfsd-metadata19:50:012
523304994617,17cyclictest34-21ksoftirqd/222:57:492
523304994613,18cyclictest34-21ksoftirqd/223:15:282
52330299469,13cyclictest1069-21runrttasks19:27:031
52330299467,16cyclictest1753-21gnome-shell00:30:231
52330299466,17cyclictest1753-21gnome-shell19:43:431
52330299466,16cyclictest1753-21gnome-shell20:15:171
52330299463,18cyclictest587453-21cstates22:10:161
523298994628,14cyclictest1-21systemd00:35:010
162399460,34rtkit-daemon2197014-21PK-Backend22:51:523
162399460,32rtkit-daemon1753-21gnome-shell20:21:593
523304994529,12cyclictest600486-21sshd22:45:392
52330499452,16cyclictest34-21ksoftirqd/200:10:012
523304994515,12cyclictest34-21ksoftirqd/220:41:442
52330299456,16cyclictest606033-21users23:00:331
52330299455,17cyclictest600648-21nm-online22:47:391
523302994518,22cyclictest538005-21grep19:50:021
5522402440,6sleep20-21swapper/220:30:112
5453302440,7sleep22197014-21PK-Backend20:10:152
52330999442,22cyclictest554736-21ntp_states20:35:263
523309994416,15cyclictest42-21ksoftirqd/323:50:273
52330499448,19cyclictest141rcu_preempt21:05:262
523304994422,18cyclictest558127-40lsb_release20:45:192
523304994418,9cyclictest34-21ksoftirqd/221:24:032
523304994418,9cyclictest34-21ksoftirqd/221:24:032
52330299445,16cyclictest591577-21sensors_temp22:20:291
523302994410,13cyclictest566434-21apt21:10:121
523302994410,12cyclictest1753-21gnome-shell20:24:101
523298994432,9cyclictest13-21ksoftirqd/023:56:390
523309994322,17cyclictest42-21ksoftirqd/300:12:193
523309994315,23cyclictest42-21ksoftirqd/300:00:283
52330999431,29cyclictest2197014-21PK-Backend20:47:423
523309994311,6cyclictest141rcu_preempt22:00:033
52330499439,17cyclictest34-21ksoftirqd/200:27:062
52330499439,11cyclictest34-21ksoftirqd/200:20:032
52330499438,17cyclictest854-21avahi-daemon21:25:582
52330499437,21cyclictest34-21ksoftirqd/222:25:282
52330499437,19cyclictest34-21ksoftirqd/223:00:162
52330499432,8cyclictest341ksoftirqd/223:39:592
523304994317,9cyclictest34-21ksoftirqd/219:15:222
523304994317,9cyclictest34-21ksoftirqd/219:15:212
52330499431,13cyclictest1753-21gnome-shell21:01:042
52330499431,13cyclictest1753-21gnome-shell21:01:042
52330499431,12cyclictest34-21ksoftirqd/223:55:022
523298994331,9cyclictest13-21ksoftirqd/023:25:270
523298994323,14cyclictest438-20systemd-journal00:00:040
52330999422,6cyclictest141rcu_preempt23:50:013
523309994225,6cyclictest42-21ksoftirqd/319:10:263
523309994225,6cyclictest42-21ksoftirqd/319:10:263
523309994213,6cyclictest141rcu_preempt20:10:133
523309994213,16cyclictest141rcu_preempt00:35:343
52330499427,14cyclictest34-21ksoftirqd/220:00:022
52330499427,14cyclictest34-21ksoftirqd/219:40:272
52330499426,19cyclictest34-21ksoftirqd/221:10:112
52330499426,13cyclictest34-21ksoftirqd/221:35:132
52330499425,17cyclictest34-21ksoftirqd/219:35:282
52330499425,17cyclictest34-21ksoftirqd/219:35:282
52330499422,9cyclictest34-21ksoftirqd/219:35:002
523302994211,15cyclictest1753-21gnome-shell22:30:431
523298994230,7cyclictest438-20systemd-journal21:19:560
523298994214,14cyclictest141rcu_preempt21:04:140
523298994214,14cyclictest141rcu_preempt21:04:140
523298994210,15cyclictest5453312cat20:10:140
162399420,5rtkit-daemon0-21swapper/221:49:032
523309994123,6cyclictest42-21ksoftirqd/300:10:023
523309994123,5cyclictest42-21ksoftirqd/320:45:013
523309994120,17cyclictest42-21ksoftirqd/323:35:493
52330999411,5cyclictest141rcu_preempt19:19:193
52330999411,5cyclictest141rcu_preempt19:19:193
52330999411,36cyclictest1753-21gnome-shell23:25:303
523309994112,11cyclictest141rcu_preempt19:35:313
523309994112,11cyclictest141rcu_preempt19:35:313
523309994111,12cyclictest141rcu_preempt21:24:593
523309994111,12cyclictest141rcu_preempt21:24:593
52330499417,17cyclictest34-21ksoftirqd/219:15:012
52330499417,17cyclictest34-21ksoftirqd/219:15:002
52330499416,13cyclictest34-21ksoftirqd/222:24:452
52330499416,10cyclictest34-21ksoftirqd/223:45:102
52330499415,9cyclictest34-21ksoftirqd/200:40:022
52330499413,8cyclictest34-21ksoftirqd/200:35:012
52330499413,19cyclictest34-21ksoftirqd/220:05:322
52330499413,11cyclictest2197014-21PK-Backend21:55:142
523304994116,8cyclictest34-21ksoftirqd/220:39:192
52330499411,21cyclictest588706-21perf22:15:002
162399410,27rtkit-daemon2197014-21PK-Backend22:42:333
52330999408,14cyclictest1358-1kworker/u9:123:20:213
523309994023,13cyclictest42-21ksoftirqd/319:54:253
523309994022,13cyclictest42-21ksoftirqd/319:24:363
52330999401,5cyclictest141rcu_preempt19:25:243
523309994012,5cyclictest141rcu_preempt23:00:153
523309994012,23cyclictest438-20systemd-journal00:26:313
523309994012,11cyclictest141rcu_preempt22:50:023
523309994011,11cyclictest141rcu_preempt21:11:413
523309994011,11cyclictest141rcu_preempt19:46:153
52330499407,10cyclictest34-21ksoftirqd/220:27:242
52330499406,9cyclictest34-21ksoftirqd/222:15:192
52330499406,18cyclictest34-21ksoftirqd/219:20:202
52330499405,11cyclictest34-21ksoftirqd/222:30:262
52330499405,10cyclictest34-21ksoftirqd/221:50:582
52330499403,9cyclictest34-21ksoftirqd/220:50:222
52330499403,9cyclictest2197014-21PK-Backend21:30:142
52330499401,19cyclictest1753-21gnome-shell22:05:132
523304994011,7cyclictest34-21ksoftirqd/221:15:492
52329899409,6cyclictest141rcu_preempt20:16:090
52329899408,5cyclictest141rcu_preempt21:45:010
523298994029,7cyclictest1753-21gnome-shell22:27:350
523298994027,8cyclictest590720-21grep22:20:140
523298994013,6cyclictest141rcu_preempt22:49:030
523298994010,14cyclictest5785952cpu21:45:140
52330999399,6cyclictest141rcu_preempt20:05:343
523309993921,7cyclictest1753-21gnome-shell21:50:323
523309993921,14cyclictest42-21ksoftirqd/322:21:423
523309993921,14cyclictest42-21ksoftirqd/320:04:283
523309993921,14cyclictest42-21ksoftirqd/320:04:283
523309993920,6cyclictest42-21ksoftirqd/321:25:233
523309993913,12cyclictest141rcu_preempt22:55:333
523309993910,15cyclictest141rcu_preempt00:30:153
52330499396,18cyclictest34-21ksoftirqd/223:20:272
52330499394,9cyclictest34-21ksoftirqd/220:22:202
52330499392,18cyclictest34-21ksoftirqd/200:20:242
523304993914,13cyclictest34-21ksoftirqd/222:44:522
52330499391,13cyclictest3242253-21nautilus20:05:022
52330499391,13cyclictest3242253-21nautilus20:05:012
523298993931,5cyclictest0-21swapper/020:08:290
523298993928,7cyclictest1069-21runrttasks19:13:020
523298993928,7cyclictest1069-21runrttasks19:13:010
52329899392,33cyclictest1753-21gnome-shell22:33:550
523298993922,13cyclictest855-21dbus-daemon21:15:010
52329899392,14cyclictest17531gnome-shell19:50:000
523298993921,14cyclictest13-21ksoftirqd/023:06:180
523298993919,15cyclictest610458-40apt-check23:15:140
523298993911,6cyclictest141rcu_preempt21:25:270
52329899391,14cyclictest17531gnome-shell19:29:590
52330999382,11cyclictest2197014-21PK-Backend23:10:273
523309993815,11cyclictest42-21ksoftirqd/300:20:123
523309993814,6cyclictest141rcu_preempt19:35:003
523309993813,6cyclictest141rcu_preempt20:55:133
523309993812,6cyclictest141rcu_preempt20:50:153
523309993811,6cyclictest141rcu_preempt23:43:233
52330999381,13cyclictest632507-21/usr/sbin/munin00:15:233
523309993810,6cyclictest141rcu_preempt20:15:273
52329899385,8cyclictest141rcu_preempt00:20:170
523298993825,8cyclictest1753-21gnome-shell22:07:190
523298993822,12cyclictest609225-21awk23:10:280
523298993822,12cyclictest549941-21apt-get20:25:020
523298993822,11cyclictest598353-21ntp_kernel_err22:40:230
52329899382,17cyclictest539398-21seq19:53:220
523298993821,13cyclictest1-21systemd23:00:010
523298993821,12cyclictest556094-21df_inode20:40:170
523298993820,13cyclictest584358-21memory22:00:220
523298993819,14cyclictest560243-21ntp_states20:50:270
523298993813,13cyclictest583695-21latency_hist22:00:030
6355562370,6sleep0161rcuc/000:25:140
5687562370,5sleep32197014-21PK-Backend21:15:243
523309993721,12cyclictest596876-21wc22:35:303
523309993714,6cyclictest141rcu_preempt21:00:273
523309993714,6cyclictest141rcu_preempt21:00:273
523309993714,5cyclictest141rcu_preempt21:35:133
523309993713,6cyclictest141rcu_preempt22:30:153
523309993713,6cyclictest141rcu_preempt20:35:023
523309993712,6cyclictest141rcu_preempt23:05:503
52329899372,8cyclictest1753-21gnome-shell00:37:230
52329899372,31cyclictest1753-21gnome-shell19:15:590
52329899372,31cyclictest1753-21gnome-shell19:15:590
52329899372,30cyclictest1753-21gnome-shell22:10:530
523298993722,11cyclictest529093-21sed19:25:010
523298993718,7cyclictest13-21ksoftirqd/000:19:550
523298993718,15cyclictest13-21ksoftirqd/019:42:350
523298993718,14cyclictest602632-21gdbus22:51:280
523298993710,6cyclictest1358-1kworker/u9:119:57:100
162399370,30rtkit-daemon0-21swapper/023:20:330
52330999369,5cyclictest141rcu_preempt23:30:223
52330999361,8cyclictest2197014-21PK-Backend22:10:273
52330999361,32cyclictest551768-21fuser20:27:283
523309993611,6cyclictest141rcu_preempt22:25:203
52329899362,9cyclictest1753-21gnome-shell21:54:200
523298993624,8cyclictest1753-21gnome-shell20:28:390
52329899362,30cyclictest554365-21df_inode20:35:160
523298993622,10cyclictest13-21ksoftirqd/022:38:480
523298993621,11cyclictest534453-21cut19:40:020
523298993621,11cyclictest534453-21cut19:40:020
523298993619,13cyclictest13-21ksoftirqd/021:23:270
523298993619,13cyclictest13-21ksoftirqd/021:23:270
523298993618,14cyclictest13-21ksoftirqd/020:30:300
523298993617,14cyclictest855-21dbus-daemon19:30:160
523309993510,6cyclictest141rcu_preempt21:45:153
523298993519,12cyclictest541723-21grep20:00:150
523298993519,12cyclictest541723-21grep20:00:150
523298993516,14cyclictest13-21ksoftirqd/023:00:210
52329899351,30cyclictest1753-21gnome-shell21:31:430
52329899351,13cyclictest0-21swapper/000:06:040
523309993412,5cyclictest141rcu_preempt19:40:273
523298993422,8cyclictest588974-21acpi22:15:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional