You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 06:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #5

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Phytec
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot5s.osadl.org (updated Thu Sep 18, 2025 00:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1351895995246,4cyclictest1417784-21kworker/u8:0+events_unbound22:22:252
1351900995037,9cyclictest391ktimers/323:35:133
1351900995026,19cyclictest0-21swapper/322:00:163
1351900994916,4cyclictest0-21swapper/323:10:273
1351900994717,26cyclictest0-21swapper/319:35:013
1351900994717,26cyclictest0-21swapper/319:35:013
1351895994734,9cyclictest311ktimers/222:30:132
1351885994741,4cyclictest0-21swapper/000:35:120
1351900994621,21cyclictest1409341-21kthreadd22:00:013
1351900994418,22cyclictest0-21swapper/322:30:263
1351895994438,4cyclictest1371353-21kworker/u8:3+events_unbound20:27:042
1351900994334,5cyclictest1440012-21grep23:25:263
1351900994235,4cyclictest1383753-21latency_hist20:40:003
1351900994216,3cyclictest0-21swapper/320:30:013
1351895994235,5cyclictest1401338-21kworker/u8:0+events_unbound21:37:152
1351895994120,7cyclictest131rcu_preempt22:10:132
1351895994120,10cyclictest131rcu_preempt23:47:212
1351885994130,7cyclictest1404678-21unixbench_singl21:40:270
1351890994032,4cyclictest1401205-21unixbench_multi21:30:281
1351900993927,7cyclictest1357129-21grep19:20:253
1351900993918,3cyclictest0-21swapper/321:30:243
1351890993932,3cyclictest0-21swapper/120:10:011
1351885993932,4cyclictest1361090-21latency_hist19:35:010
1351885993932,4cyclictest1361090-21latency_hist19:35:010
1351885993931,5cyclictest1462483-21kworker/u8:1+events_unbound00:33:540
1351885993929,7cyclictest1365163-21fschecks_count19:45:170
1351895993827,7cyclictest1455560-21grep00:10:292
1351890993831,4cyclictest1437064-21sendmail23:20:011
1351885993829,6cyclictest1420188-21fschecks_count22:30:170
1351885993827,7cyclictest0-21swapper/020:40:000
1351885993827,7cyclictest0-21swapper/020:05:260
135190099378,8cyclictest1-21systemd23:07:303
1351895993729,6cyclictest311ktimers/219:30:142
1351895993729,6cyclictest311ktimers/219:30:142
1351895993728,5cyclictest1387619-21cut20:50:212
1351890993724,8cyclictest440-21cron22:30:011
1351885993730,3cyclictest0-21swapper/022:50:000
1351885993730,3cyclictest0-21swapper/022:49:590
1351885993728,4cyclictest0-21swapper/000:03:420
135188599371,6cyclictest0-21swapper/000:10:150
1351900993627,5cyclictest1376938-21munin-plugin-st20:20:013
1351900993613,5cyclictest391ktimers/322:40:143
1351895993616,5cyclictest131rcu_preempt19:45:142
1351890993628,5cyclictest1440474-21munin-run23:30:011
1351890993628,4cyclictest0-21swapper/123:10:281
1351885993625,7cyclictest0-21swapper/020:05:010
1351885993624,8cyclictest121ktimers/020:45:120
1351885993624,7cyclictest0-21swapper/020:15:170
1351885993612,5cyclictest131rcu_preempt23:55:130
1351885993612,5cyclictest131rcu_preempt23:55:120
1351895993526,7cyclictest311ktimers/223:30:112
1351895993525,5cyclictest311ktimers/221:25:142
1351895993525,5cyclictest311ktimers/221:25:142
1351895993525,4cyclictest131rcu_preempt22:45:212
1351895993525,4cyclictest131rcu_preempt22:45:202
1351895993517,5cyclictest131rcu_preempt19:25:192
1351895993517,5cyclictest131rcu_preempt19:25:182
1351890993527,4cyclictest0-21swapper/100:10:181
1351890993527,4cyclictest0-21swapper/100:05:001
1351890993525,6cyclictest0-21swapper/123:20:161
1351885993523,8cyclictest1457319-21grep00:15:280
1351885993523,5cyclictest1435249-21kworker/u8:1+events_unbound23:15:080
1351900993426,4cyclictest637-21runrttasks20:00:153
1351900993422,8cyclictest1399969-21idleruntime-cro21:30:003
1351900993422,8cyclictest1399969-21idleruntime-cro21:29:593
1351895993427,4cyclictest0-21swapper/200:15:272
1351895993426,5cyclictest1458993-21unixbench_multi00:20:272
1351895993420,3cyclictest131rcu_preempt19:11:402
1351890993428,3cyclictest1409375-21kworker/u8:122:00:011
1351890993427,4cyclictest1369904-21munin-plugin-st20:00:001
1351890993425,6cyclictest1366053-21grep19:45:261
1351885993426,4cyclictest0-21swapper/000:30:000
1351900993325,4cyclictest0-21swapper/319:15:013
1351900993325,4cyclictest0-21swapper/300:30:183
1351900993324,5cyclictest0-21swapper/319:50:283
1351900993323,6cyclictest0-21swapper/320:50:003
1351900993323,5cyclictest0-21swapper/322:20:273
1351895993327,4cyclictest1386822-21kworker/u8:2+events_unbound20:57:092
1351895993325,4cyclictest0-21swapper/223:35:182
1351895993324,5cyclictest0-21swapper/200:05:182
1351895993314,5cyclictest1412132-21kworker/u8:2-events_freezable_power_22:28:482
135189599330,19cyclictest0-21swapper/200:00:082
1351890993325,5cyclictest1443413-21unixbench_multi23:35:271
1351890993325,4cyclictest0-21swapper/121:40:261
1351885993324,5cyclictest1425118-21sh22:44:590
1351885993323,6cyclictest0-21swapper/023:40:000
1351885993313,6cyclictest1371353-21kworker/u8:3+events_unbound20:30:140
1351885993313,6cyclictest1371353-21kworker/u8:3+events_unbound20:30:140
1351900993225,4cyclictest0-21swapper/319:30:013
1351900993225,4cyclictest0-21swapper/319:30:003
1351900993223,5cyclictest1369891-21cron20:00:013
1351900993220,8cyclictest1453662-21tail00:05:283
1351900993219,9cyclictest1430233-21idleruntime-cro23:00:003
1351895993226,4cyclictest1421260-21kworker/u8:0+events_unbound22:37:282
1351895993224,4cyclictest0-21swapper/223:40:172
1351895993216,5cyclictest131rcu_preempt21:49:102
1351890993220,7cyclictest1406383-21grep21:45:241
1351885993226,4cyclictest1450369-21kworker/u8:1+events_unbound00:08:480
135188599322,5cyclictest131rcu_preempt23:25:130
1351885993225,3cyclictest0-21swapper/021:15:170
1351885993224,4cyclictest0-21swapper/023:40:170
1351885993223,6cyclictest1412497-21unixbench_multi22:05:270
1351885993222,6cyclictest0-21swapper/021:55:000
1351885993222,6cyclictest0-21swapper/021:35:280
1351885993221,6cyclictest0-21swapper/019:10:260
1351900993124,4cyclictest1423217-21sendmail22:40:013
1351900993122,5cyclictest0-21swapper/320:55:153
1351895993120,8cyclictest311ktimers/223:05:142
1351895993118,3cyclictest131rcu_preempt20:11:302
1351895993110,6cyclictest32-21ksoftirqd/221:25:002
1351895993110,12cyclictest32-21ksoftirqd/222:55:272
1351890993123,4cyclictest0-21swapper/120:20:161
1351890993119,8cyclictest0-21swapper/122:14:591
1351890993118,9cyclictest1376954-21basename20:20:011
1351885993123,4cyclictest0-21swapper/022:50:260
1351885993122,5cyclictest0-21swapper/021:30:280
135188599312,22cyclictest0-21swapper/023:00:210
1351885993121,6cyclictest0-21swapper/021:10:230
135188599311,4cyclictest131rcu_preempt20:25:130
1351900993023,4cyclictest0-21swapper/321:45:013
1351900993021,6cyclictest1388864-21cron20:55:013
1351900993019,7cyclictest1362857-21auditctl19:40:013
135189599309,12cyclictest32-21ksoftirqd/220:19:022
135189599308,12cyclictest131rcu_preempt20:40:242
135189599307,15cyclictest32-21ksoftirqd/219:50:182
1351895993023,5cyclictest1404927-21kworker/u8:1+events_unbound22:07:232
1351895993019,3cyclictest131rcu_preempt22:15:182
1351895993017,3cyclictest131rcu_preempt23:51:572
1351895993016,3cyclictest0-21swapper/222:40:022
1351895993012,5cyclictest131rcu_preempt19:15:022
1351895993012,11cyclictest32-21ksoftirqd/222:00:012
1351895993010,13cyclictest32-21ksoftirqd/222:00:182
1351895993010,12cyclictest32-21ksoftirqd/221:00:162
1351895993010,11cyclictest32-21ksoftirqd/223:05:012
1351890993022,4cyclictest0-21swapper/120:50:201
1351890993022,4cyclictest0-21swapper/120:40:251
1351890993020,6cyclictest0-21swapper/119:25:231
1351890993020,6cyclictest0-21swapper/119:25:231
1351890993019,7cyclictest1360539-21grep19:30:261
1351890993019,7cyclictest1360539-21grep19:30:261
1351890993018,8cyclictest1-21systemd22:42:271
1351890993018,8cyclictest0-21swapper/100:00:001
1351890993018,8cyclictest0-21swapper/100:00:001
1351885993023,4cyclictest1459455-21munin-plugin-st00:25:000
1351885993020,7cyclictest0-21swapper/021:25:270
1351885993020,7cyclictest0-21swapper/021:25:260
1351900992921,5cyclictest1459001-21unixbench_multi00:20:263
135189599299,11cyclictest131rcu_preempt21:15:192
135189599298,12cyclictest32-21ksoftirqd/220:45:242
135189599294,7cyclictest32-21ksoftirqd/223:10:282
1351895992923,4cyclictest1371353-21kworker/u8:3+events_unbound20:37:052
1351895992919,5cyclictest1440034-21head23:25:272
1351895992917,4cyclictest131rcu_preempt23:20:172
1351895992912,9cyclictest131rcu_preempt20:02:002
1351895992911,3cyclictest131rcu_preempt20:05:162
1351890992922,3cyclictest0-21swapper/122:55:171
1351890992921,5cyclictest1359119-21kworker/u8:2+events_unbound19:35:171
1351890992921,4cyclictest0-21swapper/121:35:281
1351890992918,7cyclictest0-21swapper/121:10:231
135188599293,4cyclictest131rcu_preempt23:45:140
1351885992923,3cyclictest0-21swapper/019:40:000
1351885992919,6cyclictest0-21swapper/022:00:260
135190099285,4cyclictest13782012sed20:20:263
1351900992821,4cyclictest1403370-21latency_hist21:40:003
1351900992820,4cyclictest0-21swapper/322:05:273
1351900992820,4cyclictest0-21swapper/300:25:163
135189599289,4cyclictest1459666-21kworker/u8:0-writeback00:26:362
135189599289,11cyclictest32-21ksoftirqd/223:15:222
135189599288,12cyclictest32-21ksoftirqd/221:40:212
135189599287,3cyclictest131rcu_preempt00:35:092
135189599286,13cyclictest32-21ksoftirqd/219:35:252
1351895992820,6cyclictest1424931-21kworker/u8:0+events_unbound22:50:142
1351895992820,4cyclictest0-21swapper/219:45:012
1351895992816,5cyclictest131rcu_preempt20:23:232
1351895992810,12cyclictest131rcu_preempt21:10:392
1351890992821,4cyclictest0-21swapper/122:10:001
1351890992818,6cyclictest1355847-21idleruntime-cro19:20:011
1351890992817,6cyclictest0-21swapper/120:27:011
1351890992815,10cyclictest1351877-21cyclictest23:30:111
135188599282,4cyclictest131rcu_preempt22:10:140
1351885992822,4cyclictest1412132-21kworker/u8:2+events_unbound22:17:230
1351885992818,6cyclictest0-21swapper/023:20:270
135188599281,4cyclictest131rcu_preempt21:00:130
1351885992811,14cyclictest1388889-21idleruntime-cro20:55:010
1351900992720,4cyclictest0-21swapper/323:30:223
1351900992714,10cyclictest1464767-21run-parts00:40:003
135189599279,11cyclictest32-21ksoftirqd/223:55:182
135189599279,11cyclictest32-21ksoftirqd/223:55:182
135189599276,13cyclictest32-21ksoftirqd/219:20:152
135189599275,6cyclictest32-21ksoftirqd/200:30:272
1351895992720,5cyclictest1404927-21kworker/u8:1+events_unbound21:52:192
1351895992720,5cyclictest1371353-21kworker/u8:3+events_unbound20:32:102
1351895992720,5cyclictest1371353-21kworker/u8:3+events_unbound20:32:102
1351895992711,2cyclictest0-21swapper/221:33:062
1351890992721,4cyclictest1386822-21kworker/u8:2+events_unbound20:55:431
1351890992720,3cyclictest0-21swapper/119:15:011
1351885992720,5cyclictest1401338-21kworker/u8:0+events_unbound21:49:250
1351885992719,4cyclictest0-21swapper/022:40:010
1351885992717,6cyclictest0-21swapper/019:15:270
135188599270,9cyclictest11-21ksoftirqd/022:25:130
1351900992620,3cyclictest0-21swapper/322:10:243
1351900992620,3cyclictest0-21swapper/320:10:173
1351900992618,4cyclictest0-21swapper/319:16:453
1351900992615,7cyclictest0-21swapper/323:40:253
1351895992610,6cyclictest32-21ksoftirqd/219:55:252
1351890992619,3cyclictest0-21swapper/122:35:181
1351890992618,5cyclictest0-21swapper/100:25:161
1351890992618,4cyclictest0-21swapper/121:20:151
1351890992618,4cyclictest0-21swapper/100:20:271
1351890992616,7cyclictest1387173-21idleruntime-cro20:49:591
135188599263,3cyclictest1429204-21meminfo22:55:210
135188599262,4cyclictest131rcu_preempt19:55:120
1351885992620,4cyclictest1417784-21kworker/u8:0+events_unbound22:24:330
1351885992619,4cyclictest0-21swapper/019:25:230
1351885992619,4cyclictest0-21swapper/019:25:230
1351885992618,5cyclictest1435353-21idleruntime-cro23:15:010
1351900992519,3cyclictest0-21swapper/323:50:153
1351900992519,3cyclictest0-21swapper/321:10:153
1351900992519,3cyclictest0-21swapper/300:00:183
1351900992518,3cyclictest0-21swapper/300:15:003
1351900992517,4cyclictest0-21swapper/323:20:013
1351900992517,4cyclictest0-21swapper/321:55:003
1351900992516,6cyclictest0-21swapper/320:10:003
1351900992515,6cyclictest0-21swapper/323:00:183
135190099251,4cyclictest13642622wc19:40:243
135189599257,10cyclictest131rcu_preempt21:08:392
1351890992519,3cyclictest0-21swapper/121:55:011
1351890992518,3cyclictest0-21swapper/123:40:271
1351890992517,3cyclictest0-21swapper/100:35:241
1351890992514,8cyclictest231ktimers/121:00:131
135188599258,10cyclictest11-21ksoftirqd/021:20:220
1351885992515,3cyclictest0-21swapper/020:40:020
135188599251,21cyclictest1432231-21awk23:05:140
135188599251,20cyclictest0-21swapper/023:30:440
1351900992418,4cyclictest1378441-21kworker/u8:1+events_unbound20:42:163
1351900992416,4cyclictest0-21swapper/322:50:263
135189099249,4cyclictest131rcu_preempt20:00:121
1351890992418,4cyclictest1424931-21kworker/u8:0+events_unbound22:52:301
1351890992418,3cyclictest0-21swapper/100:15:191
1351890992417,3cyclictest0-21swapper/119:50:181
1351890992416,5cyclictest1374794-21unixbench_singl20:10:271
1351890992416,4cyclictest0-21swapper/122:30:161
1351890992416,4cyclictest0-21swapper/119:25:011
135188599248,9cyclictest11-21ksoftirqd/023:50:270
135188599247,10cyclictest11-21ksoftirqd/020:55:200
135188599245,5cyclictest131rcu_preempt20:10:130
1351885992417,5cyclictest1376698-21kworker/u8:0+events_unbound20:20:400
1351885992417,5cyclictest1360900-21kworker/u8:0+events_unbound19:40:200
135188599241,6cyclictest14083922ntpq21:55:170
1351900992317,3cyclictest0-21swapper/322:25:253
1351900992317,3cyclictest0-21swapper/321:00:113
1351900992315,5cyclictest0-21swapper/321:45:183
1351900992314,5cyclictest440-21cron23:50:003
1351890992316,3cyclictest0-21swapper/122:20:281
1351890992314,5cyclictest0-21swapper/123:00:181
1351890992313,7cyclictest1425860-21cat22:45:201
1351890992313,7cyclictest1425860-21cat22:45:201
1351890992311,8cyclictest231ktimers/121:25:131
1351890992311,8cyclictest231ktimers/121:25:131
135188599236,10cyclictest11-21ksoftirqd/019:20:190
1351885992318,3cyclictest0-21swapper/019:50:140
135188599230,19cyclictest0-21swapper/021:07:050
1351900992216,3cyclictest0-21swapper/320:30:233
1351900992216,3cyclictest0-21swapper/320:30:223
1351900992215,4cyclictest391ktimers/321:17:373
1351900992213,6cyclictest391ktimers/322:15:183
1351900992211,8cyclictest1393162-21ssh21:10:013
1351890992214,4cyclictest0-21swapper/119:40:171
1351890992212,7cyclictest1415810-21timerwakeupswit22:15:251
135189099220,4cyclictest14465072grep23:45:251
1351900992113,5cyclictest1450559-21sshd23:58:333
1351900992113,5cyclictest1450559-21sshd23:58:323
1351890992110,8cyclictest1395649-21ntp_states21:15:221
1351890992110,8cyclictest1393201-21cut21:10:011
1351900992015,2cyclictest0-21swapper/319:45:173
1351900992013,3cyclictest440-21cron00:20:003
1351900992011,6cyclictest0-21swapper/322:45:153
1351900992011,6cyclictest0-21swapper/322:45:143
1351900992010,7cyclictest1437529-21tune2fs23:20:183
135189099206,3cyclictest0-21swapper/120:38:051
1351890992013,4cyclictest1432176-21/usr/sbin/munin23:05:241
1351890992012,4cyclictest0-21swapper/120:35:001
1351890992012,4cyclictest0-21swapper/120:35:001
1351890992011,6cyclictest0-21swapper/100:30:161
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional